Stream switching among compressed video streams coded at different quality levels and bit rates is examined, and two enhanced stream switching schemes for H.264 video are proposed in this work. The flexibility of the original H.264 stream switching scheme is achieved at the cost of coding efficiency by introducing primary and secondary SP/SI pictures. In contrast, no modification is made to the original compressed streams in our enhanced schemes so that coding efficiency is maintained at switching points. When switching occurs, the two new schemes use a new picture type, called the difference picture (DIFF), in different ways to compensate the mismatch of reference frames. The difference picture can be coded efficiently, and it is easy to pick a good quantization step size to meet the quality requirement. It is shown by experimental results that the two new schemes outperform the original H.264 switching scheme in the sense that they can achieve prompt stream switching without noticeable quality degradation, and a small amount of bit rate overhead is demanded only when switching occurs.
In this work, we improve the H.264 error resilient coding scheme in [1, 2] with a hybrid scheme that generates alternative SP macroblocks utilizing both multiple reference frames  and the concealed versions of corrupted frames . The new scheme is more robust and able to work effectively under different coding environments. Although the bit rate overhead introduced by  and  is acceptable for some applications, we use an adaptive coding and bit stream replacement mechanism to reduce the overhead furthermore to meet the strict bandwidth constraints. Specifically, two different versions of alternative SP macroblocks are coded using different quantization levels. They provide different levels of error resilient performance with different bit rate consumptions. When the sender attempts to replace the originally coded version of a target macroblock in the bit stream, it will select one proper version according to the importance of the macroblock. The importance of the macroblock is measured by its influence to the subjective quality of the current frame and its impact to subsequent frames. The implementation and the standard conformance of the proposed scheme are detailed in this work.
A novel H.264 error resilient scheme aiming at stopping or reducing error propagation using the SP/SI coded macroblocks is proposed to maintain the quality of transmitted video through erroneous channels in this work. Specifically, for each encoded macroblock, we encode additional predicted versions using different reference frames (or different prediction methods) and save them as SP/SI macroblocks. During transmission, these SP/SI macroblocks are used to replace the originally coded macroblocks in the output video stream if they are affected by previous errors detected by the receiver. The way to encode these SP/SI macroblocks ensures that such a replacement will not cause any mismatch at the decoder side. It is confirmed by experimental results that the proposed scheme is effective in reducing error propagation so as to enhance the error resilient capability of H.264 video. This scheme introduces a small amount of overhead in the bit rate only when there are transmission errors, and does not have an overhead when no error occurs.
As emerging video coding standards, e.g. H.264, aim at high-quality video contents at low bit-rates, the encoding and decoding processes require much more computation than most existing standards do. This paper analyzes software implementation of a real-time H.264 decoder on general-purpose processors with media instructions. Specifically, we discuss how to optimize the speed of H.264 decoders on Intel Pentium 4 processors. This paper first analyzes the reference implementation to identify the time-consuming modules. Our study shows that a number of components, e.g., motion compensation and inverse integer transform, are the most time-consuming modules in the H.264 decoder. Second, we present a list of performance optimization methods using media instructions to improve the efficiency of these modules. After appropriate optimizations, the decoder speed improved by more than 3x---it can decode a 720×480 resolution video sequence at 48 frames per second on 2.4GHz Intel Pentium 4 processors compared to reference software’s 12 frames per second. The optimization techniques demonstrated in this paper can also be applied to other video/image processing applications. Additionally, after presenting detailed application behavior on general-purpose processors, this paper discusses a few recommendations on how to design future efficient/powerful video/image applications/standards with given hardware implications.