Group-of-pictures-based unequal error protection for scalable video coding extension of H.264/AVC

Jun Xu; Xiaokang Yang; Shibao Zheng; Li Song

doi:10.1117/1.3152774

1 June 2009 Group-of-pictures-based unequal error protection for scalable video coding extension of H.264/AVC

Jun Xu, Xiaokang Yang, Shibao Zheng, Li Song

Author Affiliations +

Optical Engineering, Vol. 48, Issue 6, 060502 (June 2009). https://doi.org/10.1117/1.3152774

Abstract

We address the problem of unequal error protection (UEP) for SNR enhancement layer network abstraction layer (NAL) units of scalable video coding extension of H.264/AVC standard over wireless packet-erasure channel. We develop a UEP scheme by jointly selecting SNR NAL units and allocating unequal amounts of protection to selected NAL units for every group of pictures in the sequence. A simple heuristic algorithm is proposed to quickly derive the protection pattern. Experimental results demonstrate the proposed UEP scheme provides significant error resilience.

1. Introduction

Scalable video coding (SVC) combined with unequal error protection (UEP) is promising to deliver video robustly. SVC extension of the H.264/AVC standard^{1, 2} has achieved significant improvement in coding efficiency when providing spatial, temporal, and SNR scalability in a single bitstream relative to the scalable profiles of prior video coding standards. Medium-granular SNR scalability (MGS) provides network abstraction layer (NAL) unit-based SNR scalability.

A UEP scheme for bitstream transmission over the packet erasure channel roughly comprises the source selection stage and channel rate allocation stage. Which part of the encoded bitstream is decided for transmission in the source stage selection. Given the source rate and channel rate, appropriate protection is assigned for different parts of the source rate in the channel rate allocation stage to maximize overall distortion reduction at decoder. A traditional one-dimensional UEP scheme jointly considering source selection and channel rate allocation on image or video bitstream is investigated in which channel rate assignment has to be nondecreasing or nonincreasing.^{3, 4, 5} The recent progress is two-dimensional UEP for SVC with a combined temporal and SNR scalability.^{6, 7} The similar packetization scheme for SVC extension of H.264/AVC standard is considered in Refs. 8, 9. Reference 8 presents a fast channel rate assignment algorithm, but Refs. 6, 7, 8 are only for the channel rate allocation stage. Reference 9 takes the source selection stage into account but is designed for fine granular SNR scalability, which is already removed from the standard. The bitstream of the base layer is involved in the UEP optimization in Refs. 6, 7, 8, 9.

We develop a UEP scheme for MGS enhancement NAL units over packet erasure channel jointly considering source selection stage and channel rate allocation stage with a simple heuristic solution. In this paper, bitstream of the base layer is assumed delivered to the receiver error free.

2. Unequal Error Protection

Reed–Solomon codes are used to generate the forward error correction (FEC) codes. The packetization scheme is like the scheme in Refs. 6, 7, as depicted by Fig. 1. The length and height of the FEC codes for the MGS NAL unit $(i, j)$ in frame $i$ and SNR layer $j$ are denoted by $K (i, j)$ and $h (i, j)$ , respectively, where $i = 0, \dots, G - 1$ , $j = 0, \dots, F - 1$ . $G$ is the group-of-pictures (GOP) length in frame, and $F$ is the number of SNR layers. We protect the source bitstream for every GOP in the sequence. The length of the MGS NAL unit $(i, j)$ is $N - K (i, j)$ . $N$ is the number of transmission packets. The transmission packet size is denoted by $L$ . $N L$ is the transmission block target rate.

Fig. 1

UEP scheme for MGS enhancement NAL units in a GOP.

We want to find the best allocation such that the overall distortion reduction is maximized

Eq. 1

D_{reduction} = \sum_{i = 0}^{G - 1} \sum_{j = 0}^{F - 1} Δ D (i, j) P (i, j),

where

Δ D (i, j)

is the distortion reduction from the inclusion of MGS NAL unit

(i, j)

, which can be readily calculated by the method in Ref. 2.

P (i, j)

is the probability of correctly receiving the MGS NAL unit

(i, j)

. A two-state Markov model is used to approximate the wireless channel’s packet loss behavior.

P (m, N)

is the probability of losing

m

packets within

N

packets, which is calculated by the Markov model. Thus,

Eq. 2

P (i, j) = \sum_{m = 0}^{K (i, j)} P (m, N) .

Now, the objective of the optimization is to find the proper channel rate allocation matrix

K

,

Eq. 3

K = [\begin{matrix} K (0, 0) & \dots & K (0, F - 1) \\ ⋮ & ⋱ & ⋮ \\ K (G - 1, 0) & \dots & K (G - 1, F - 1) \end{matrix}], - 1 ⩽ K (i, j) ⩽ N - 1 .

Here, channel rate allocation

K (i, j) = - 1

means the corresponding MGS NAL unit is not to transmit. Thus,

- 1 \leq K (i, j) \leq N - 1

integrates the source selection stage and channel rate allocation stage. Constraint 1 is

Eq. 4

\sum_{i = 0}^{G - 1} \sum_{j = 0}^{F - 1} h (i, j) ⩽ L,

which restricts the total amount of source bits and channel bits not to exceed the transmission block target rate.

h (i, j)

is calculated as

Eq. 5

h (i, j) = {\begin{cases} ⌈ \frac{B (i, j)}{N - K (i, j)} ⌉ & 0 ⩽ K (i, j) ⩽ N - 1 \\ 0 & K (i, j) = - 1 . \end{cases}

B (i, j)

is the number of source data bytes in MGS NAL unit

(i, j)

. The constraint 2 is

Eq. 6

K (i, j) ⩾ K (i, j + 1), i = 0, \dots, G - 1, j = 0, \dots, F - 2,

because a higher layer MGS NAL unit will be useless if the lower one is not available. Now the optimization problem is then expressed as

Eq. 7

\max D_{reduction} (K) s.t. (4), (6) .

We introduce $G F (N + 1)$ binary variables $Q (i, j, q)$ ( $i = 0, \dots$ , $G - 1$ , $j = 0, \dots$ , $F - 1$ , $q = 0, \dots, N$ ) to reformulate the primal problem 7 inspired by Ref. 10. Let $N + 1$ binary variables represent $K (i, j)$ for frame $i$ and SNR layer $j$ . Set the $q$ ’th binary variable as 1 and other binary variables as 0 to represent channel rate allocation $K (i, j) = q - 1$ . Thus

Eq. 8

P (i, j, q) = \sum_{m = 0}^{q - 1} P (m, N), (q = 0, \dots, N) .

Let

b (i, j, q) = [Δ D (i, j)] [P (i, j, q)]

,

b (i, j, q) = 0

for

q = 0

.

Eq. 9

h (i, j, q) = {\begin{cases} ⌈ \frac{B (i, j)}{N - q + 1} ⌉ & 1 ⩽ q ⩽ N \\ 0 & q = 0 \end{cases} .

Hence,

b (i, j, q)

and

h (i, j, q)

are increasing with

q

for given

i, j

and they could be seemed as the profit and weight of item

q

on table

i, j

, respectively.

L

is regarded as weight capacity of knapsack. Then the problem is to select one and only one item from each table in order to maximize overall profit with the constraint 6 and without exceeding the knapsack’s weight capacity. This is like a multiple-choice knapsack problem (MCKP), which is well known to be nodeterministic polynomial-time (NP) hard. We construct a heuristic solution as follows:

Step 1: Sort all items except items with

q = 0

by their profit density

b (i, j, q) ∕ h (i, j, q)

(

i = 0, \dots, G - 1

,

j = 0, \dots, F - 1

,

q = 1, \dots, N

) in decreasing order. Denote it as list1. Every element list1

[x]

contains the item index

(i, j, q)

, profit

b

and weight

h

,

b

and

h

represent the corresponding

b (i, j, q)

and

h (i, j, q)

, respectively.

Set

K (i, j) = - 1, \forall i, j

; Set

C = 0

;

For (

f = 0

;

f < F

;

f + +

)

Set

x = 0

;

While (

list 1 [x] . j = = f

&&

C + list 1 [x] . h \Leftarrow L

&&

x < G \cdot F \cdot N

)

If

(K (list 1 [x] . i, list 1 [x] . j) = = - 1

&& (

list 1 [x] . j = = 0

∥ (list 1 [x] . j > 0

&&

list 1 [x] . q \Leftarrow

K (list 1 [x] . i

,

list 1 [x] . j - 1) + 1)

))

K (list 1 [x] . i, list 1 [x] . j) = list 1 [x] . q - 1

;

C = C + list 1 [x] . h

;

End If

x + +

;

End While

End For

Step 2: Sort

λ_{i, j} = (b (i, j, K (i, j) + 2) - b (i, j, K (i, j) + 1)) ∕ (h (i, j, K (i, j) + 2) - h (i, j, K (i, j) + 1)), \forall i, j

in decreasing order. Denote it as list2. Every element

list 2 [y]

contains the item index

(i, j)

.

Set

found = 0

; Set

y = 0

;

While (

C + h (list 2 [y] . i

,

list 2 [y] . j

,

K (list 2 [y] . i

,

list 2 [y] . j) + 2) - h (list 2 [y] . i

,

list 2 [y] . j, K (list 2 [y] . i, list 2 [y] . j) + 1) \Leftarrow L

&&

found = = 0

&&

y < G \cdot F)

If (

list 2 [y] . j = = 0

∥ (list 2 [y] . j > 0

&&

K (list 2 [y] . i

,

list 2 [y] . j) + 1 \Leftarrow

K (list 2 [y] . i, list 2 [y] . j - 1)

))

K (list 2 [y] . i, list 2 [y] . j) = K (list 2 [y] . i, list 2 [y] . j) + 1

;

C = C + h (list 2 [y] . i

,

list 2 [y] . j

,

K (list 2 [y] . i

,

list 2 [y] . j) + 2) - h (list 2 [y] . i

,

list 2 [y] . j, K (list 2 [y] . i, list 2 [y] . j) + 1)

;

found = 1

;

End If

y + +

;

End While

if

(found = = 1)

goto Step 2;

else End.

The complexity of algorithm mainly comes from sorting algorithm, which is $O [(G F N) \log (G F N)]$ .

3. Experimental Results

We encode with $G = 16$ and $F = 3$ MGS enhancement layers. One-hundred different runs of the experiments were conducted to transmit video sequences with different packet-loss patterns. For the two-state Markov channel, the average burst length is 9.57. We set $N = 100$ . “Foreman 200” means the result for sequence foreman and $L = 200$ .

The heuristic channel rate allocation algorithm runs very fast and on average produces results within $0.1 s$ on a PC with $2.0 - GHz$ CPU. We compare the UEP scheme with EEP scheme in which the elements of $K$ have the same value under different average packet loss rate (PLR) and the results are depicted in Fig. 2. Figure 2 shows the result of EEP without any MGS enhancement NAL unit transmission when source rate to transmit exceeds transmission block target rate. It can be seen that proposed UEP scheme shows graceful degradation and an improvement of at least $2.5 dB$ in quality is achieved when compared with equal error protection (EEP). Figure 2 shows the performance of proposed UEP and EEP when source rate to transmit is less than transmission block target rate. At most, $3 - dB$ gains are achieved for proposed UEP over EEP. If all MGS enhancement NAL units have to be included in the transmission block (as in Refs. 6, 7) then the Eq. 7 changes slightly by setting $0 ⩽ K (i, j) ⩽ N - 1$ . It can be solved by starting from $K (i, j) = 0$ and applying step 2 in proposed heuristic solution. Curves “UEP-all” in Fig. 2 show it is inferior to the proposed UEP scheme especially at high packet loss rate.

Fig. 2

PSNR comparison of the UEP scheme against EEP scheme for MGS enhancement NAL units.

We take a subjective test of some results according to the double-stimulus continuous quality-scale method suggested by ITU-R BT.500-10.¹¹ The mean opinion scores are rescaled to a range of 0–100. Difference mean opinion scores (DMOS) are calculated as the difference between the original video and the test video. Table 1 shows the DMOS of UEP and EEP at $PLR = 30 %$ . It can be seen that subjective rating of the proposed method is better than the others.

Table 1

DMOS comparison of the UEP scheme against EEP scheme at PLR=30%

Sequence L	EEP	Proposed	UEP all	Sequence L	EEP	Proposed
Harbour 1100	39.82	36.25	37.58	Harbour 600	40.11	36.72
Mobile 1200	40.42	36.09	38.91	Mobile 800	41.59	38.78
Foreman 600	39.13	30.74	34.02	Foreman 200	39.98	33.79
Crew 1000	40.15	31.88	36.54	Crew 500	40.51	36.15

4. Conclusions

We propose a UEP scheme by jointly selecting source packets and allocating unequal amounts of protection to MGS enhancement NAL units for every GOP. A heuristic algorithm is proposed to quickly get the solution. As illustrated by our simulation results, the proposed UEP scheme provides significant error resilience.

Acknowledgments

This work was supported, in part, by NSFC (Grants No. 60502034 and No. 60625103), Hi-Tech Research and Development Program of China 863 (Grant No. 2006AA01Z124), Grant No. NCET-06-0409, and the 111 Project.

References

1.

H. Schwarz, D. Marpe, and T. Wiegand, “Overview of the scalable video coding extension of the H.264/AVC standard,” IEEE Trans. Circuits Syst. Video Technol., 17 (9), 1103 –1120 (2007). https://doi.org/10.1109/TCSVT.2007.905532 1051-8215 Google Scholar

2.

I. Amonou, N. Cammas, S. Kervadec, and S. Pateux, “Optimized rate distortion extraction with quality layers in the scalable extension of H.264 AVC,” IEEE Trans. Circuits Syst. Video Technol., 17 (9), 1186 –1193 (2007). https://doi.org/10.1109/TCSVT.2007.906870 1051-8215 Google Scholar

3.

A. Mohr, E. A. Riskin, and R. E. Ladner, “Unequal loss protection: graceful degradation of image quality over packet erasure channels through forward error correction,” IEEE J. Sel. Areas Commun., 18 (6), 819 –828 (2000). https://doi.org/10.1109/49.848236 0733-8716 Google Scholar

4.

X. K. Yang, C. Zhu, Z. G. Li, X. Lin, and N. Ling, “An unequal packet loss resilience scheme for video over the internet,” IEEE Trans. Multimedia, 7 (4), 753 –765 (2005). https://doi.org/10.1109/TMM.2005.846782 1520-9210 Google Scholar

5.

H. Cai, B. Zeng, G. Shen, Z. Xiong, and S. Li, “Error-resilient unequal error protection of fine granularity scalable video bitstreams,” EURASIP J. Appl. Signal Process., 1 –11 (2006). https://doi.org/10.1155/ASP/2006/45412 1110-8657 Google Scholar

6.

T. Fang and L. P. Chau, “GOP-based channel rate allocation using genetic algorithm for scalable video streaming over error-prone networks,” IEEE Trans. Image Process., 15 (6), 1323 –1330 (2006). https://doi.org/10.1109/TIP.2005.864159 1057-7149 Google Scholar

7.

Y. Wang, T. Fang, L. P. Chau, and K. H. Yap, “Two-dimensional channel coding scheme for MCTF-based scalable video coding,” IEEE Trans. Multimedia, 9 (1), 37 –45 (2007). https://doi.org/10.1109/TMM.2006.886339 1520-9210 Google Scholar

8.

H. Ha and C. Yim, “Layer-weighted unequal error protection for scalable video coding extension of H.264/AVC,” IEEE Trans. Consum. Electron., 54 (2), 736 –744 (2008). https://doi.org/10.1109/TCE.2008.4560155 0098-3063 Google Scholar

9.

A. Naghdinezhad, M. R. Hashemi, and O. Fatemi, “A novel adaptive unequal error protection method for scalable video over wireless networks,” 1 –6 (2007). Google Scholar

10.

A. E. Mohr, “Bit allocation in sub-linear time and the multiple-choice knapsack problem,” 352 –361 (2002). Google Scholar

11.

International Telecommunications Union–Radiocommunication Sector (ITU-R), “Methodology for the subjective assessment of the quality of television pictures,” (2000). Google Scholar

Citation Download Citation

Jun Xu, Xiaokang Yang, Shibao Zheng, and Li Song "Group-of-pictures-based unequal error protection for scalable video coding extension of H.264/AVC," Optical Engineering 48(6), 060502 (1 June 2009). https://doi.org/10.1117/1.3152774

Published: 1 June 2009

Access the abstract

JOURNAL ARTICLE
3 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 6 scholarly publications.

Explore citations on Lens.org

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Magnesium

Scalable video coding

Signal to noise ratio

Video

Binary data

Chemical elements

Distortion

1.

Introduction

2.

Unequal Error Protection