1 June 2009 Group-of-pictures-based unequal error protection for scalable video coding extension of H.264/AVC
Author Affiliations +
We address the problem of unequal error protection (UEP) for SNR enhancement layer network abstraction layer (NAL) units of scalable video coding extension of H.264/AVC standard over wireless packet-erasure channel. We develop a UEP scheme by jointly selecting SNR NAL units and allocating unequal amounts of protection to selected NAL units for every group of pictures in the sequence. A simple heuristic algorithm is proposed to quickly derive the protection pattern. Experimental results demonstrate the proposed UEP scheme provides significant error resilience.
Xu, Yang, Zheng, and Song: Group-of-pictures-based unequal error protection for scalable video coding extension of H.264/AVC



Scalable video coding (SVC) combined with unequal error protection (UEP) is promising to deliver video robustly. SVC extension of the H.264/AVC standard1, 2 has achieved significant improvement in coding efficiency when providing spatial, temporal, and SNR scalability in a single bitstream relative to the scalable profiles of prior video coding standards. Medium-granular SNR scalability (MGS) provides network abstraction layer (NAL) unit-based SNR scalability.

A UEP scheme for bitstream transmission over the packet erasure channel roughly comprises the source selection stage and channel rate allocation stage. Which part of the encoded bitstream is decided for transmission in the source stage selection. Given the source rate and channel rate, appropriate protection is assigned for different parts of the source rate in the channel rate allocation stage to maximize overall distortion reduction at decoder. A traditional one-dimensional UEP scheme jointly considering source selection and channel rate allocation on image or video bitstream is investigated in which channel rate assignment has to be nondecreasing or nonincreasing.3, 4, 5 The recent progress is two-dimensional UEP for SVC with a combined temporal and SNR scalability.6, 7 The similar packetization scheme for SVC extension of H.264/AVC standard is considered in Refs. 8, 9. Reference 8 presents a fast channel rate assignment algorithm, but Refs. 6, 7, 8 are only for the channel rate allocation stage. Reference 9 takes the source selection stage into account but is designed for fine granular SNR scalability, which is already removed from the standard. The bitstream of the base layer is involved in the UEP optimization in Refs. 6, 7, 8, 9.

We develop a UEP scheme for MGS enhancement NAL units over packet erasure channel jointly considering source selection stage and channel rate allocation stage with a simple heuristic solution. In this paper, bitstream of the base layer is assumed delivered to the receiver error free.


Unequal Error Protection

Reed–Solomon codes are used to generate the forward error correction (FEC) codes. The packetization scheme is like the scheme in Refs. 6, 7, as depicted by Fig. 1. The length and height of the FEC codes for the MGS NAL unit (i,j) in frame i and SNR layer j are denoted by K(i,j) and h(i,j) , respectively, where i=0,,G1 , j=0,,F1 . G is the group-of-pictures (GOP) length in frame, and F is the number of SNR layers. We protect the source bitstream for every GOP in the sequence. The length of the MGS NAL unit (i,j) is NK(i,j) . N is the number of transmission packets. The transmission packet size is denoted by L . NL is the transmission block target rate.

Fig. 1

UEP scheme for MGS enhancement NAL units in a GOP.


We want to find the best allocation such that the overall distortion reduction is maximized


where ΔD(i,j) is the distortion reduction from the inclusion of MGS NAL unit (i,j) , which can be readily calculated by the method in Ref. 2. P(i,j) is the probability of correctly receiving the MGS NAL unit (i,j) . A two-state Markov model is used to approximate the wireless channel’s packet loss behavior. P(m,N) is the probability of losing m packets within N packets, which is calculated by the Markov model. Thus,


Now, the objective of the optimization is to find the proper channel rate allocation matrix K ,


Here, channel rate allocation K(i,j)=1 means the corresponding MGS NAL unit is not to transmit. Thus, 1K(i,j)N1 integrates the source selection stage and channel rate allocation stage. Constraint 1 is


which restricts the total amount of source bits and channel bits not to exceed the transmission block target rate. h(i,j) is calculated as


B(i,j) is the number of source data bytes in MGS NAL unit (i,j) . The constraint 2 is


because a higher layer MGS NAL unit will be useless if the lower one is not available. Now the optimization problem is then expressed as



We introduce GF(N+1) binary variables Q(i,j,q) ( i=0, , G1 , j=0, , F1 , q=0,,N ) to reformulate the primal problem 7 inspired by Ref. 10. Let N+1 binary variables represent K(i,j) for frame i and SNR layer j . Set the q ’th binary variable as 1 and other binary variables as 0 to represent channel rate allocation K(i,j)=q1 . Thus


Let b(i,j,q)=[ΔD(i,j)][P(i,j,q)] , b(i,j,q)=0 for q=0 .


Hence, b(i,j,q) and h(i,j,q) are increasing with q for given i,j and they could be seemed as the profit and weight of item q on table i,j , respectively. L is regarded as weight capacity of knapsack. Then the problem is to select one and only one item from each table in order to maximize overall profit with the constraint 6 and without exceeding the knapsack’s weight capacity. This is like a multiple-choice knapsack problem (MCKP), which is well known to be nodeterministic polynomial-time (NP) hard. We construct a heuristic solution as follows:

Step 1: Sort all items except items with q=0 by their profit density b(i,j,q)h(i,j,q) ( i=0,,G1 , j=0,,F1 , q=1,,N ) in decreasing order. Denote it as list1. Every element list1 [x] contains the item index (i,j,q) , profit b and weight h , b and h represent the corresponding b(i,j,q) and h(i,j,q) , respectively.
Set K(i,j)=1,i,j ; Set C=0 ;
For ( f=0 ; f<F ; f++ )
 Set x=0 ;
 While ( list1[x].j==f && C+list1[x].hL && x<GFN )
  If (K(list1[x].i,list1[x].j)==1) && ( list1[x].j==0 (list1[x].j> 0) &&
    list1[x].q K(list1[x].i) , ((list1[x].j1)+1) ))
    K(list1[x].i,list1[x].j)=list1[x].q1 ;
    C=C+list1[x].h ;
  End If
   x++ ;
 End While
End For
Step 2: Sort λi,j=(b(i,j,K(i,j)+2)b(i,j,K(i,j)+1))(h(i,j,K(i,j)+2)h(i,j,K(i,j)+1)),i,j in decreasing order. Denote it as list2. Every element list2[y] contains the item index (i,j) .
Set found=0 ; Set y=0 ;
While ( C+h(list2[y].i) , list2[y].j , K(list2[y].i) , ((list2[y].j)+2)h(list2[y].i) , (list2[y].j,K(list2[y].i,list2[y].j)+1)L && found==0 && (y<GF)
 If ( list2[y].j==0 (list2[y].j> 0) && K(list2[y].i) , (list2[y].j)+1 (K(list2[y].i,list2[y].j1) ))
   K(list2[y].i,list2[y].j)=K(list2[y].i,list2[y].j)+1 ;
   C=C+h(list2[y].i) , list2[y].j , K(list2[y].i) , ((list2[y].j)+2)h(list2[y].i) , (list2[y].j,K(list2[y].i,list2[y].j)+1) ;
   found=1 ;
 End If
y++ ;
End While
if (found==1) goto Step 2;
else End.

The complexity of algorithm mainly comes from sorting algorithm, which is O[(GFN)log(GFN)] .


Experimental Results

We encode with G=16 and F=3 MGS enhancement layers. One-hundred different runs of the experiments were conducted to transmit video sequences with different packet-loss patterns. For the two-state Markov channel, the average burst length is 9.57. We set N=100 . “Foreman 200” means the result for sequence foreman and L=200 .

The heuristic channel rate allocation algorithm runs very fast and on average produces results within 0.1s on a PC with 2.0-GHz CPU. We compare the UEP scheme with EEP scheme in which the elements of K have the same value under different average packet loss rate (PLR) and the results are depicted in Fig. 2. Figure 2 shows the result of EEP without any MGS enhancement NAL unit transmission when source rate to transmit exceeds transmission block target rate. It can be seen that proposed UEP scheme shows graceful degradation and an improvement of at least 2.5dB in quality is achieved when compared with equal error protection (EEP). Figure 2 shows the performance of proposed UEP and EEP when source rate to transmit is less than transmission block target rate. At most, 3-dB gains are achieved for proposed UEP over EEP. If all MGS enhancement NAL units have to be included in the transmission block (as in Refs. 6, 7) then the Eq. 7 changes slightly by setting 0K(i,j)N1 . It can be solved by starting from K(i,j)=0 and applying step 2 in proposed heuristic solution. Curves “UEP-all” in Fig. 2 show it is inferior to the proposed UEP scheme especially at high packet loss rate.

Fig. 2

PSNR comparison of the UEP scheme against EEP scheme for MGS enhancement NAL units.


We take a subjective test of some results according to the double-stimulus continuous quality-scale method suggested by ITU-R BT.500-10.11 The mean opinion scores are rescaled to a range of 0–100. Difference mean opinion scores (DMOS) are calculated as the difference between the original video and the test video. Table 1 shows the DMOS of UEP and EEP at PLR=30% . It can be seen that subjective rating of the proposed method is better than the others.

Table 1

DMOS comparison of the UEP scheme against EEP scheme at PLR=30%

Sequence LEEPProposedUEP allSequence LEEPProposed
Harbour 110039.8236.2537.58Harbour 60040.1136.72
Mobile 120040.4236.0938.91Mobile 80041.5938.78
Foreman 60039.1330.7434.02Foreman 20039.9833.79
Crew 100040.1531.8836.54Crew 50040.5136.15



We propose a UEP scheme by jointly selecting source packets and allocating unequal amounts of protection to MGS enhancement NAL units for every GOP. A heuristic algorithm is proposed to quickly get the solution. As illustrated by our simulation results, the proposed UEP scheme provides significant error resilience.


This work was supported, in part, by NSFC (Grants No. 60502034 and No. 60625103), Hi-Tech Research and Development Program of China 863 (Grant No. 2006AA01Z124), Grant No. NCET-06-0409, and the 111 Project.


1.  H. Schwarz, D. Marpe, and T. Wiegand, “Overview of the scalable video coding extension of the H.264/AVC standard,” IEEE Trans. Circuits Syst. Video Technol.1051-8215 17(9), 1103–1120 (2007). 10.1109/TCSVT.2007.905532 Google Scholar

2.  I. Amonou, N. Cammas, S. Kervadec, and S. Pateux, “Optimized rate distortion extraction with quality layers in the scalable extension of H.264 AVC,” IEEE Trans. Circuits Syst. Video Technol.1051-8215 17(9), 1186–1193 (2007). 10.1109/TCSVT.2007.906870 Google Scholar

3.  A. Mohr, E. A. Riskin, and R. E. Ladner, “Unequal loss protection: graceful degradation of image quality over packet erasure channels through forward error correction,” IEEE J. Sel. Areas Commun.0733-8716 18(6), 819–828 (2000). 10.1109/49.848236 Google Scholar

4.  X. K. Yang, C. Zhu, Z. G. Li, X. Lin, and N. Ling, “An unequal packet loss resilience scheme for video over the internet,” IEEE Trans. Multimedia1520-9210 7(4), 753–765 (2005). 10.1109/TMM.2005.846782 Google Scholar

5.  H. Cai, B. Zeng, G. Shen, Z. Xiong, and S. Li, “Error-resilient unequal error protection of fine granularity scalable video bitstreams,” EURASIP J. Appl. Signal Process.1110-8657 1–11 (2006). 10.1155/ASP/2006/45412 Google Scholar

6.  T. Fang and L. P. Chau, “GOP-based channel rate allocation using genetic algorithm for scalable video streaming over error-prone networks,” IEEE Trans. Image Process.1057-7149 15(6), 1323–1330 (2006). 10.1109/TIP.2005.864159 Google Scholar

7.  Y. Wang, T. Fang, L. P. Chau, and K. H. Yap, “Two-dimensional channel coding scheme for MCTF-based scalable video coding,” IEEE Trans. Multimedia1520-9210 9(1), 37–45 (2007). 10.1109/TMM.2006.886339 Google Scholar

8.  H. Ha and C. Yim, “Layer-weighted unequal error protection for scalable video coding extension of H.264/AVC,” IEEE Trans. Consum. Electron.0098-3063 54(2), 736–744 (2008). 10.1109/TCE.2008.4560155 Google Scholar

9.  A. Naghdinezhad, M. R. Hashemi, and O. Fatemi, “A novel adaptive unequal error protection method for scalable video over wireless networks,” in Proc. Int. Symp. on Consumer Electronics (ISCE 2007), 20–23 June 2007, Dallas, TX, pp. 1–6, IEEE, Piscataway, NJ (2007). Google Scholar

10.  A. E. Mohr, “Bit allocation in sub-linear time and the multiple-choice knapsack problem,” in Proc. Data Compression Conf. (DCC 2002), 2–4 April 2002, Snowbird, UT, pp. 352–361, IEEE Computer Society, Washington, DC (2002). Google Scholar

11. International Telecommunications Union–Radiocommunication Sector (ITU-R), “Methodology for the subjective assessment of the quality of television pictures,” ITU-R Rec. BT. 500–10, Std. (2000). Google Scholar

Jun Xu, Xiaokang Yang, Shibao Zheng, Li Song, "Group-of-pictures-based unequal error protection for scalable video coding extension of H.264/AVC," Optical Engineering 48(6), 060502 (1 June 2009). https://doi.org/10.1117/1.3152774

Back to Top