1.

INTRODUCTION

In recent years, the growth in delivery of video at scale for broadcast and streaming applications (from Netflix, YouTube, Disney etc.) has inspired further research into content-adaptive transcoding. The goal is to deliver high-quality content at progressively lower bitrates by adapting the transcoder for each input at a fine-grained level of control. In 2013, YouTube was the first to adopt this strategy for its User-Generated-Content (UGC) by building a pipeline that is based on clip popularity by re-processing a clip with an enhanced pre-processor in combination with a different built-in transcoder. Around the same time, Netflix’s seminal work on perclip and per-shot encoding⁴ for High Value Content (HVC) videos showed that an exhaustive search of the coding parameter space can lead to significant gains in Rate-Distortion (RD) tradeoffs per clip. These gains offset the high one-time computational cost of encoding as the same encoded clip may be streamed to millions of viewers across many different Content Delivery Networks (CDNs),⁵ thus effectively saving bandwidth and network resources. That idea has since been revisited and has become more efficient by applying the Viterbi algorithm across shots and parameter spaces.⁶ Over the past years a lot of researchers have focused on the optimisation of a high-level parameter (target bitrate or quantisation factor or objective quality) to generate an optimal bitrate ladder for a clip as part of a Adaptive Bitrate Streaming (ABR).^7–9

In our previous work,¹⁰ we showed that the RD tradeoff can be directly addressed by applying a numerical optimisation scheme to estimate the appropriate Lagrangian λ multiplier for a given clip (see Section 2) for standard dynamic range (SDR) videos. We observed an average BD-rate improvement of 1.9% for HEVC, 1.3% for VP9,¹¹ and 0.5% in AV1.¹² In our latest work,¹² we further demonstrated that additional BD-rate(%) gains, from 0.5% to 4.9%, for AV1 could be achieved by adopting a per frame-type optimisation.

In this paper, we explore the idea of λ optimisation on High-Dynamic Range (HDR)/Wide Color Gamut (WCG) material.¹³ HDR/WCG systems can capture, process, and reproduce a scene conveying the full range of perceptible shadow and highlight details beyond normal dynamic range (SDR) video systems.

Similarly to our latest work,¹² we propose a content-adaptive transcoder optimisation at a global and a deeper frame-type level. The core new ideas are i) consideration of RD Optimisation (RDO) for HDR content, ii) optimisation of the RDO parameters on a frame-hierarchy basis, and iii) investigation of various convergence criteria that result in the minimisation of the computational load. Our experiments in Section 5 demonstrate that a frame-based tuning of video encoder can lead to an average gain of 1.63% of BD-rate (best recorded gain of 9.3%) compared to the standardised method. Moreover, the average gains per shot range between 0.58 and 3.43% for HDR video content in AV1.

Section 2 gives an overview of previous research work and λ definition in rate control. Section 3 explains the proposed methodology as well as the multi-dimensional optimisation. Section 4 then details the experimental set-up including the test sequences, the keyframe combination selection, the framework implementation. Section 5 reports on the experimental findings.

2.

BACKGROUND

The work of Sullivan et al¹⁴ laid the foundations for optimising the RD tradeoff in modern video codecs. By taking a Lagrange multiplier approach the joint optimisation problem is posed as the minimisation of J = D + λR, where λ is the Lagrange multiplier controlling the tradeoff. This idea is the basis of the RDO process used especially in making mode decisions in modern codecs. The independent variable in this optimisation is usually qp, a quantiser step size. Increasing qp reduces rate R but increases distortion D. Also, a different choice of λ yields different R,D pairs.

Different codecs devised different recipes to derive the optimal λ value for RDO through an empirical relationship with qp. In libaom-AV1¹ λ is empirically related to q_i (Quantizer Index: ≈ qp * 4 in the AV1 codebase), as follows:

where A is a constant depending on the frame type (3.2 ≤ A ≤ 3.3) and q_dc = f (q_i, A) is defined through a discrete valued Look Up Table (LUT) (0 ≤ q_i ≤ 255 for AV1). This λ−qp relationship is not necessarily optimal for a particular clip because the empirical relationship was derived for optimality over an entire test corpus. To maximise gains, λ should be content dependent.

Per Clip λ Optimisation. The idea of adapting λ based on video content is not entirely new. Zhang and Bull¹⁵ altered λ based on distortion statistics on a frame-basis for HEVC. In our previous work,^{10, 11, 16} we introduced the idea of an adaptive λ on a clip basis, using a single modified λ = kλ_o across all the frames in a clip. Here, λ_o represents the default value deployed from the relevant empirical relationship e.g. Eq. (1). In order to find the optimal λ value, we deployed numerical optimisers (Brent’s method and Golden-search¹⁷) that minimised the BD-rate as a cost function. Later work,¹⁰ considered the use of Machine Learning techniques to reduce the required computational load.

Per Clip, Per Frame-Type λ Optimisation. In our latest work,¹² we showed that this method of global λ tuning yields average BD-rate(%) gains of only 0.539% and 0.097% for AV1 and HEVC, respectively. These modest improvements are probably due to the fact that current modern video encoders are content-adaptive by nature and include many new improvements such as partition tools, Inter/Intra prediction tools and modern hierarchical reference frame-structure. Another important aspect to consider is that the heuristics and empirical shortcuts used in this content-adaptive implementation of the Lagrangian parameter, deviates from classical RDO theory, which normally requires λ to be constant across the sequences over which distortion is measured. For example in Eq.(1) λ changes with frame type. Motivated by this observation, in our latest work,¹² we studied the effect of isolating the optimisation of λ for different frame types. Results on SDR sequences showed that optimising λ purely for Keyframes (KF), Golden-Frames (GF), Alternate-reference Frames (ARF), leads to average BD-rate gains of 4.92% compared to global λ optimisation (only 0.54% BR-Rate gains).

HDR in AV1. The quality of compressed 4K and 8K HDR content was evaluated by Pourazad et. al.,¹⁸ drone content from P. Topiwala et.al.,¹⁹ and gaming (Nabajeet et. al.²⁰). More relevant here is the work of Zhou et. al²¹ for HEVC that expresses distortion D in terms of the HDR-VDP-2 quality metric.²² They presented an algorithm for prediction of λ at the CTU level which resulted in 5% BD-rate improvement w.r.t. a reference implementation of HEVC (HM16.19).

3.

DIRECT λ OPTIMISATION IN AV1

As noted in Section 2, the encoder is determining λ independently on a frame basis inside the encoder. In this work, we explore the impact of treating λ optimisation as a multi variable search problem at a frame basis w.r.t. HDR content. The following sections expand our main strategies for the direct optimisation of the λ parameter.

BD-rate Optimisation. For the n^th RDO decision in clip m, we propose that λ_n = k_nλ₀. We estimate k = [k₁, …, k_N] (where we assume N RDO decisions) to maximise the BD-rate gain using MS-SSIM²³ as the quality metric (Q_m). The cost function C_m(k) can therefore be formulated as:

where R_m(kλ_o, Q) is the bitrate of the m^th clip at quality Q, using λ = kλ_o for the N RDO decisions and Q₁,Q₂ defined as usual.³ R_m(·, ·) is derived from the MS-SSIM-based RD curve generated using P qp measurements. Here we use P = 5: qp ={27, 39, 49, 59, 63}.

The flow of the optimisation framework is reported in Algorithm 1. We can see that repeated computations of the BD-rate are required, and this incurs a huge computational cost. To address this, we deploy the idea of proxies for parameter selection as proposed by Ping H. et. al.^{24, 25} They observe that using different speed settings of the encoder at the same target quality/quantizer level results primarily in bitrate differences that are directly proportional to the content complexity (see section 5.1). That idea also extends to the use of lower resolution proxies. Therefore we can reduce computational load by performing optimisation using faster encoder presets ‘em and lower resolution proxies.

Multi-Dimensional Optimiser. Our previous study for finding the optimal λ multiplier in AV1 used Brent’s line search method.¹² The focus was on applying a modifier k to only one sub-module in the encoder. However, here we explore the scenario of a multiple dimensional search for finding the optimal λ for multiple frame types each associated with a different k. For this multi-dimensional search, we have various options like Nelder-Mead Simplex,²⁶ Conjugate Gradient,²⁷ Powell’s method,²⁸ etc.

In order to select a suitable optimiser, we conducted a simple analysis by carrying out an exhaustive grid-search on a single video to study the surface of our objective function. For this study, we chose the clip NocturneRoom from the AOM Common-Testing-Configuration (AOM-CTC)²⁹ set, and optimised λ for two different frametypes: λ_KF for the KFs and λ_GF/ARF, for the GF/ARFs. For all other frame-types, λ was set to the default. The grid search range for both λ_KF and λ_GF/ARF is 0.6 to 5.4, in steps of 0.1. This provided 2,401 anchor points resulting in a total of 12,005 RD points for analysis.

Figure 1 shows the contour plot of the BD-rate % (MS-SSIM) objective function for a sample clip. The surface is clearly smooth, and a gradient based method is expected to converge to a sub-optimal solution (local minimum) due to very low gradient. This observation was confirmed by testing gradient based methods such as the Nelder-Mead Simplex method²⁶ and the Conjugate Gradient.²⁷ Both of these methods converged erroneously after the first iteration as the gradient was very close to zero. Therefore, we explored line-search constrained methods. One of the best performing was the modified Powell method,²⁸ which succeeded to reach the global minima for the test clip (see red-lines on the Figure 1).

Figure 1:

Plot of the optimisation surface, where X axis represents the λ_KF [K1] multiplier value and Y axis the λ_GF/ARF [K2] value, contour is C_m(k₁, …, k_n). The red line shows the variation of K1 and K2 values when Powell’s method²⁸ is deployed for direct optimisation.

4.

EXPERIMENTAL SETUP

4.1

4K HDR Corpus

For our experimental studies, we formed a video corpus consisting of 50 video clips (6500 frames) curated from various public sources. All the videos are normalized to BT.2020 color primaries with SMPTE2084 Perceptual Quantizer (PQ) transfer function and represented in the YUV colourspace inside YUV-Y4M containers. All the conversions and normalization of the clips are implemented with HDRTools.³⁰ The configuration file for the conversion to YUV Space with PQ Signal is available in our project page^∗. The 50 clips contain 130 frames, a resolution of 3840x2160/4096x2160, and can be further grouped into 7 shot groups. Figure 2 illustrates sample frames of the dataset and Table 1 gives a short description of the content of these 7 shot groups. More information on the dataset, including computed Spatial and Temporal Information (SI and TI)³¹ and Dynamic Range (DR) can be found in our project page^∗.

Figure 2:

Sample frames from the corpus.

Table 1:

High-level description of the shots.

Shot Group	Description
Cosmos32	Vibrant animated sequence, high temporal complexity, 24fps
Meridian29, 32	Natural sequence, high spatial complexity, 59.94fps
Nocturne29, 32	Natural sequence, medium spatial complexity, 60fps
Sol Levante29, 32	Animated sequence, medium-high temporal complexity, 24fps
Sparks29, 32	Sequence with medium motion and wide dynamic range, 59.94fps
SVT33	Very high natural complexity sequences, 50fps
Cables 4K34	Outdoor sequences with moderate complexity, 59.94fps

5.

EXPERIMENTS & RESULTS

5.2

Multiple Frame Types λ Optimisation Performance

One of the objectives of this work is to compare different combinations of frame-type dependent λ optimisations with the ultimate aim to find the best λ optimisation strategy. First, we considered 4 modes, as in our previous work.¹² Particularly, we set λ = kλ_o for some grouped frame types, and kept λ = λ_o for all other frame types. These four modes are: i) All Frames, which refers to the global λ tuning method where we set the same multiplier k for all frames, as previously proposed by Ringis et al.,¹¹ ii) KF, which refers to the scenario where we set λ = kλ_o tune for KF, iii) GF-ARF, which refers to λ optimisation for GF and ARF frames, iv) KF-GF-ARF, which refers to optimising λ for KF, GF, and ARF frames.

In addition to the above four modes, we introduced a new search method Powell [KF-GF/ARF], which is a multidimensional joint search, where we deploy two k values, with k₁ for KF and k₂ for GF and ARF frames. Thus, λ_KF = k₁λ_o and λ_GF/ARF = k₂λ_o, and for others we use the default λ value.

Table 3 reports the BD-rate (MS-SSIM %), MS-SSIM (dB), VMAF³⁹ gains for these different optimisation strategies for the whole corpus which is divided into 7 different shots groups (see Table 1). The underlined result shows the highest gains in terms of BD-rate that the proposed tuning brings compared to (4K S2). The results presented are averaged on a shot group basis. Also, the minimum and average BD-rate in the shot group are recorded. Another perspective of the resulting BD-rates from all tested clips is illustrated in the histograms of the BD-rate values in Figure 3.

Figure 3:

Histogram distribution of BD-rate(%) MS-SSIM for various frame-level tuning methods.

First, we observe that the new values of k are, on average, significantly different from k = 1, hence verifying our initial hypothesis of a “better” λ value. Inspecting the Brent optimiser results from both Table 3 and Figure 3, it transpires that the KF-GF-ARF method is achieving the best BD-rate gains on average (1.12%). Analysing the results on per-shot basis, we were able to see significant improvements in BD-rates for certain shots. Best average gains recorded were for Sol-Levante, where the BD-rate improved from 0.79% to 2.94%, and this was followed by Cosmos with an improvement from 1.44% to 2.74%.

Another important observation is that the multidimensional search method Powell [KF, GF/ARF], appears to be the overall top performer, with average BD-rate gains of 1.63%. The histogram in Figure 3e also shows that the improvement is consistent for the majority of the videos. This also evident in terms of bitrate, as we achieve a bitrate reduction of 0.64% with KF, GF, ARF and 4.25% with Powell [KF, GF/ARF] at QP39. We also observed that for this improved bitrate savings, we have a minimal loss of MS-SSIM with 0.06dB and 0.35 for VMAF on average.

From Table 3, we notice that clips within the same shot group respond differently. These deviations can be partly explained by variations in spatial and temporal information (See project page^∗). Analysing the distribution of results obtained with the Powell method, we observed that the higher BD-rate gains (> 4%) were acquired for clips with high spatial information/complexity (5 clips). Videos with lower temporal information, exhibited higher gains. Videos with low temporal and low spatial complexity had no improvements. Different optimisation modes exhibited very different distributions, for instance, in the classic tuning method, the BD-rate improvements were noticed with clips with SI between 400-500 while for tuning mode of λ for KF/GF gave large loss for the same range of videos.

6.

CONCLUSION

We have presented a new method of per-clip optimisation based on the rate control λ multiplier in AV1 for different frame types. The proposed method was tested on a 4K-HDR corpus of 50 videos. We reported improved average BD-rate gains of 1.6% for the proposed per-frame-type per-clip optimisation compared to the 0.4% for the global per-clip λ-optimisation. The proposed method showed improvements up to 3.4% on average for certain shots compared to the method of global λ tuning. The best improvements in BD-rate range from 2.1% to 9.3% with our proposed method. We have also showed that the computational complexity of the optimisation process can be mitigated by employing proxy settings and restricting the number of iterations used by the Powell optimiser. Thus, our proposed method of multivariable Powell-based optimiser gave the best improvements on average. Future work will focus on exploring the current implementation of deriving λ from the quantiser along with more in-depth analysis of the quality aspect of the results including a subjective study.