1.

Introduction

H.264/AVC is a state of the art video coding standard with a heavy computational load (CL), which affects encoder performance in many real-time applications. Several papers reducing the computational load have been published, and it is one of the important issues on H.264/AVC.¹

The reference software for H.264/AVC includes a function that encodes an interframe in intramode. This feature is especially useful for frames involving an abrupt scene change (ASC), because the ASC frame is difficult to correlate with the previous frames. To achieve higher video quality and lower fluctuation in video quality, an appropriate rate control (RC) is necessary for an ASC frame.² This requires the detection of an ASC before the ASC frame is encoded.

If a distinct difference between consecutive frames can be detected by a suitable dissimilarity metric (DM), an ASC can be declared whenever the DM exceeds a given threshold. Various such DMs have been published and can largely be categorized into two groups: those designed for use with compressed video³ and those designed for use with uncompressed video.^{4, 5} Since an ASC detection for RC cannot be performed before encoding the current frame, DMs for compressed video are not suitable. Therefore, only DMs for uncompressed video are considered in this work, which are further categorized into two subgroups: pixel-based and histogram-based DMs. Pixel-based DMs are simpler, but are inferior when analyzing high-motion video or when dealing with partial changes within a whole frame, such as flashes and objects that appear abruptly. To detect ASCs robustly, the well-known mean square error (MSE) between two consecutive frames is suggested as a DM.⁴ This DM is defined as follows:

2.

Proposed Detection Algorithm

Most DMs for uncompressed video use the pixels of the previous frame to verify how strongly the consecutive frames correlate. This process demands an increased CL to save the previous frame. If instead we can obtain information about correlation in the process of H.264 encoding, the CL can be reduced. In Ref. 2, the predicted peak signal-to-noise ratio (PPSNR) is presented for use in the H.264 RC. The PPSNR is calculated using the current picture and the previously reconstructed frame. Since this reconstructed frame has already been saved in the process of H.264 encoding, PPSNR can be computed without increasing the CL.

As discussed in Sec. 1, pixel-based DMs are defective when applied to high-motion video and partial frame changes. To fix these problems while minimizing the CL, we divide a whole frame into several parts. Figure 1 illustrates how to divide a whole frame into 12 parts. Based on these ideas, we propose a new DM for frame $i$ and part $x$ as follows:

Fig. 1

Frame divided into 12 parts to improve tolerance of partial changes.

3.

Experimental Results

We evaluated the proposed algorithm along with Refs. 4, 5 for three high-motion sequences of QCIF size: “Goal of the Tournament for 2006 Worldcup” (Worldcup), “60 Greatest Playoff Moments for NBA” (NBA), and “Final Fantasy X-2” (FF-X2). Further information about these test sequences is tabulated in Table 1. The proposed algorithm was implemented on JM9.8⁶ to use the previously saved reconstructed frame in the H.264 encoding process. The other algorithms were also implemented on JM9.8 to allow CL comparisons within the same platform. The parameters for all tests were set as follows: only the first frame within a group of pictures was encoded in intramode; there were no $B$ frames and no frame skip; the search range was 16; the number of reference frames was one; rate distortion optimization was enabled; the entropy coding was context adaptive binary arithmetic coding; the buffer size was set to half of the target bit rates and the frame rate was $30\mathrm{fps}$ . The target bit rates were set to 57.6, 115.2, and $230.4\mathrm{kbps}$ to verify their effect on the proposed algorithm.

Table 1

Video sequences for performance evaluation.

Table 2 summarizes the detection performance of the tested algorithms as measured by the number of False and Miss detections. A False detection indicates that an ASC is declared, but does not occur. Contrarily, a Miss detection indicates that an ASC occurs, but is not declared. The thresholds of the proposed algorithm were set after intensive experiments: $\alpha =0.75$ , $N_f=12$ , and $\beta =0.70$ . The thresholds of the other tested algorithms were modified to improve performance for the test sequences as follows: the threshold of Eq. 1 in Ref. 4 was set to 1450, and the threshold of absolute difference frame variance (ADFV) after histogram equalization with normalization (HEN) in Ref. 5 was decreased from 50 to 30. As shown in Table 2, the proposed algorithm executed fewer Miss detections than the existing algorithms. Regarding False detections, the proposed algorithm was superior to Ref. 4 in all test sequences and equivalent to Ref. 5 in Worldcup. In NBA and FF-X2, the proposed algorithm executed more False detections than Ref. 5. This occurred by virtue of HEN because the two sequences, especially NBA, contain many frames involving flashes. However, HEN requires a very high CL, which seriously obstructs the real-time H.264 RC.

Table 2

Comparison results of False and Miss detections between proposed and existing algorithms. See Refs. 4, 5.

When a False detection occurs, bits are allocated unnecessarily. This causes an increase in the subsequent quantization parameters and a decrease in video quality. When a Miss detection occurs, bits are allocated inappropriately. In this case, the distortion propagates to the subsequent frames.⁷ Based on these observations, the sum of False and Miss detections is a better parameter for evaluating an ASC detection algorithm in viewpoint of H.264 RC. Using the sum of the False and Miss detections, we define a parameter indicating the detection performance numerically as follows:

To compare the CL of the tested algorithms, we used $\mathrm{Intel}\circledR$ ${\mathrm{VTune}}^{™}$ Performance Analyzer 8.0 after optimizing the CL of Refs. 4, 5 in C code level. When computing a variance of Eq. 1 in Ref. 4, we obtained a squared value from memory without multiplication after rounding a mean value to the nearest integer. In Ref. 5, although a noncandidate ASC frame can be filtered using the mean absolute frame difference (MAFD), we cannot predict whether an ASC will occur in the next frame. Therefore, HEN must be executed in every frame and the equalized pixels must be saved. Since this process requires a high CL, we instead saved the original pixels of frames $\left(i\text{-}1\right)$ and $\left(i\text{-}2\right)$ to perform HEN if necessary for frame $i$ . To further reduce the CL, all pixels of a frame were copied with a width of one frame when saved to memory.

Figure 2 shows the CLs of the three tested algorithms. For comparison, the CL of the frame-layer H.264 RC is also included. We set the $\mathrm{Intel}\circledR$ ${\mathrm{VTune}}^{™}$ to the time-based mode using the operating system (OS) timer. The sampling interval was one millisecond. For corroboration, all tests were performed on three computers: $\mathrm{Intel}\circledR$ ${\mathrm{Core}}^{™}2$ CPU 6400 at $2.13\mathrm{GHz}$ , $\mathrm{Intel}\circledR$ $\mathrm{Pentium}\circledR$ D CPU $3.00\mathrm{GHz}$ , and $\mathrm{Intel}\circledR$ $\mathrm{Pentium}\circledR$ D CPU $3.40\mathrm{GHz}$ . The OS for all three computers was $\mathrm{Microsoft}\circledR$ $\mathrm{Windows}\circledR$ XP. The timer samples were obtained by each computer and for each sequence. The timer samples shown in Fig. 2 are the sum of the samples measured by each computer. On summing timer samples over three computers and three sequences, the proposed algorithm achieved a CL 37.1% lower than that of Ref. 4 and 92.1% lower than that of Ref. 5.

Fig. 2

Timer samples presenting the computational load of each abrupt scene change detection algorithm.

4.

Conclusion

We propose a cost-effective ASC detection algorithm, because the existing ASC detection algorithms either perform poorly or result in heavy CL that seriously obstruct the real-time H.264 RC for very high-motion video. Compared with a pixel-based algorithm, the detection performance of the proposed algorithm increases by 66%, while the CL is reduced by 37%. Compared to an algorithm using both pixel-based and histogram-based DMs, the detection performance of the proposed algorithm is 23% lower. However, the CL is dramatically reduced by 92%. Taking both the detection performance and CL into consideration, the proposed algorithm is more suitable than the existing algorithms for detecting ASCs in real-time H.264 RC.