## 1.

## Introduction

Scalable video coding (SVC) combined with unequal error protection (UEP) is promising to deliver video robustly. SVC extension of the H.264/AVC standard^{1, 2} has achieved significant improvement in coding efficiency when providing spatial, temporal, and SNR scalability in a single bitstream relative to the scalable profiles of prior video coding standards. Medium-granular SNR scalability (MGS) provides network abstraction layer (NAL) unit-based SNR scalability.

A UEP scheme for bitstream transmission over the packet erasure channel roughly comprises the source selection stage and channel rate allocation stage. Which part of the encoded bitstream is decided for transmission in the source stage selection. Given the source rate and channel rate, appropriate protection is assigned for different parts of the source rate in the channel rate allocation stage to maximize overall distortion reduction at decoder. A traditional one-dimensional UEP scheme jointly considering source selection and channel rate allocation on image or video bitstream is investigated in which channel rate assignment has to be nondecreasing or nonincreasing.^{3, 4, 5} The recent progress is two-dimensional UEP for SVC with a combined temporal and SNR scalability.^{6, 7} The similar packetization scheme for SVC extension of H.264/AVC standard is considered in Refs. 8, 9. Reference 8 presents a fast channel rate assignment algorithm, but Refs. 6, 7, 8 are only for the channel rate allocation stage. Reference 9 takes the source selection stage into account but is designed for fine granular SNR scalability, which is already removed from the standard. The bitstream of the base layer is involved in the UEP optimization in Refs. 6, 7, 8, 9.

We develop a UEP scheme for MGS enhancement NAL units over packet erasure channel jointly considering source selection stage and channel rate allocation stage with a simple heuristic solution. In this paper, bitstream of the base layer is assumed delivered to the receiver error free.

## 2.

## Unequal Error Protection

Reed–Solomon codes are used to generate the forward error correction (FEC) codes. The packetization scheme is like the scheme in Refs. 6, 7, as depicted by Fig. 1. The length and height of the FEC codes for the MGS NAL unit $(i,j)$ in frame $i$ and SNR layer $j$ are denoted by $K(i,j)$ and $h(i,j)$ , respectively, where $i=0,\dots ,G-1$ , $j=0,\dots ,F-1$ . $G$ is the group-of-pictures (GOP) length in frame, and $F$ is the number of SNR layers. We protect the source bitstream for every GOP in the sequence. The length of the MGS NAL unit $(i,j)$ is $N-K(i,j)$ . $N$ is the number of transmission packets. The transmission packet size is denoted by $L$ . $NL$ is the transmission block target rate.

We want to find the best allocation such that the overall distortion reduction is maximized

where $\Delta D(i,j)$ is the distortion reduction from the inclusion of MGS NAL unit $(i,j)$ , which can be readily calculated by the method in Ref. 2. $P(i,j)$ is the probability of correctly receiving the MGS NAL unit $(i,j)$ . A two-state Markov model is used to approximate the wireless channel’s packet loss behavior. $P(m,N)$ is the probability of losing $m$ packets within $N$ packets, which is calculated by the Markov model. Thus,Now, the objective of the optimization is to find the proper channel rate allocation matrix $K$ ,## Eq. 3

$$K=\left[\begin{array}{ccc}K(0,0)& \cdots & K(0,F-1)\\ \vdots & \ddots & \vdots \\ K(G-1,0)& \cdots & K(G-1,F-1)\end{array}\right],\phantom{\rule{1em}{0ex}}-1\u2a7dK(i,j)\u2a7dN-1.$$## Eq. 5

$$h(i,j)=\{\begin{array}{ll}\lceil \frac{B(i,j)}{N-K(i,j)}\rceil & 0\u2a7dK(i,j)\u2a7dN-1\\ 0& K(i,j)=-1.\end{array}\phantom{\}}$$## Eq. 6

$$K(i,j)\u2a7eK(i,j+1),\phantom{\rule{1em}{0ex}}i=0,\dots ,G-1,\phantom{\rule{1em}{0ex}}j=0,\dots ,F-2,$$## Eq. 7

$$\mathrm{max}\phantom{\rule{0.2em}{0ex}}{D}_{\text{reduction}}\left(K\right)\phantom{\rule{1em}{0ex}}\mathit{s.t.}\phantom{\rule{1em}{0ex}}\left(4\right),\left(6\right).$$We introduce $GF(N+1)$ binary variables $Q(i,j,q)$ ( $i=0,\dots $ , $G-1$ , $j=0,\dots $ , $F-1$ , $q=0,\dots ,N$ ) to reformulate the primal problem 7 inspired by Ref. 10. Let $N+1$ binary variables represent $K(i,j)$ for frame $i$ and SNR layer $j$ . Set the $q$ ’th binary variable as 1 and other binary variables as 0 to represent channel rate allocation $K(i,j)=q-1$ . Thus

Let $b(i,j,q)=\left[\Delta D(i,j)\right]\left[P(i,j,q)\right]$ , $b(i,j,q)=0$ for $q=0$ .## Eq. 9

$$h(i,j,q)=\{\begin{array}{ll}\lceil \frac{B(i,j)}{N-q+1}\rceil & 1\u2a7dq\u2a7dN\\ 0& q=0\end{array}\phantom{\}}.$$

Step 1: Sort all items except items with $q=0$ by their profit density $b(i,j,q)\u2215h(i,j,q)$ ( $i=0,\cdots ,G-1$ , $j=0,\cdots ,F-1$ , $q=1,\cdots ,N$ ) in decreasing order. Denote it as list1. Every element list1 $\left[x\right]$ contains the item index $(i,j,q)$ , profit $b$ and weight $h$ , $b$ and $h$ represent the corresponding $b(i,j,q)$ and $h(i,j,q)$ , respectively. |

Set $K(i,j)=-1,\forall i,j$ ; Set $C=0$ ; |

For ( $f=0$ ; $f<F$ ; $f++$ ) |

Set $x=0$ ; |

While ( $\text{list}1\left[x\right].j==f$ && $C+\text{list}1\left[x\right].h\Leftarrow L$ && $x<G\cdot F\cdot N$ ) |

If $(K(\text{list}1\left[x\right].i,\text{list}1\left[x\right].j)==-1\phantom{)}$ && ( $\text{list}1\left[x\right].j==0$ $\parallel (\text{list}1\left[x\right].j>0\phantom{)}$ && |

$\text{list}1\left[x\right].q\Leftarrow $ $K(\text{list}1\left[x\right].i\phantom{)}$ , $\phantom{(}\phantom{(}\text{list}1\left[x\right].j-1)+1)$ )) |

$K(\text{list}1\left[x\right].i,\text{list}1\left[x\right].j)=\text{list}1\left[x\right].q-1$ ; |

$C=C+\text{list}1\left[x\right].h$ ; |

End If |

$x++$ ; |

End While |

End For |

Step 2: Sort ${\lambda}_{i,j}=(b(i,j,K(i,j)+2)-b(i,j,K(i,j)+1))\u2215(h(i,j,K(i,j)+2)-h(i,j,K(i,j)+1)),\forall i,j$ in decreasing order. Denote it as list2. Every element $\text{list}2\left[y\right]$ contains the item index $(i,j)$ . |

Set $\text{found}=0$ ; Set $y=0$ ; |

While ( $C+h(\text{list}2\left[y\right].i\phantom{)}$ , $\text{list}2\left[y\right].j$ , $K(\text{list}2\left[y\right].i\phantom{)}$ , $\phantom{(}\phantom{(}\text{list}2\left[y\right].j)+2)-h(\text{list}2\left[y\right].i\phantom{)}$ , $\phantom{(}\text{list}2\left[y\right].j,K(\text{list}2\left[y\right].i,\text{list}2\left[y\right].j)+1)\Leftarrow L$ && $\text{found}==0$ && $\phantom{(}y<G\cdot F)$ |

If ( $\text{list}2\left[y\right].j==0$ $\parallel (\text{list}2\left[y\right].j>0\phantom{)}$ && $K(\text{list}2\left[y\right].i\phantom{)}$ , $\phantom{(}\text{list}2\left[y\right].j)+1\Leftarrow $ $\phantom{(}K(\text{list}2\left[y\right].i,\text{list}2\left[y\right].j-1)$ )) |

$K(\text{list}2\left[y\right].i,\text{list}2\left[y\right].j)=K(\text{list}2\left[y\right].i,\text{list}2\left[y\right].j)+1$ ; |

$C=C+h(\text{list}2\left[y\right].i\phantom{)}$ , $\text{list}2\left[y\right].j$ , $K(\text{list}2\left[y\right].i\phantom{)}$ , $\phantom{(}\phantom{(}\text{list}2\left[y\right].j)+2)-h(\text{list}2\left[y\right].i\phantom{)}$ , $\phantom{(}\text{list}2\left[y\right].j,K(\text{list}2\left[y\right].i,\text{list}2\left[y\right].j)+1)$ ; |

$\text{found}=1$ ; |

End If |

$y++$ ; |

End While |

if $(\text{found}==1)$ goto Step 2; |

else End. |

The complexity of algorithm mainly comes from sorting algorithm, which is $O\left[\left(GFN\right)\mathrm{log}\left(GFN\right)\right]$ .

## 3.

## Experimental Results

We encode with $G=16$ and $F=3$ MGS enhancement layers. One-hundred different runs of the experiments were conducted to transmit video sequences with different packet-loss patterns. For the two-state Markov channel, the average burst length is 9.57. We set $N=100$ . “Foreman 200” means the result for sequence foreman and $L=200$ .

The heuristic channel rate allocation algorithm runs very fast and on average produces results within $0.1\phantom{\rule{0.3em}{0ex}}\mathrm{s}$ on a PC with $2.0\text{-}\mathrm{GHz}$ CPU. We compare the UEP scheme with EEP scheme in which the elements of $K$ have the same value under different average packet loss rate (PLR) and the results are depicted in Fig. 2. Figure 2 shows the result of EEP without any MGS enhancement NAL unit transmission when source rate to transmit exceeds transmission block target rate. It can be seen that proposed UEP scheme shows graceful degradation and an improvement of at least $2.5\phantom{\rule{0.3em}{0ex}}\mathrm{dB}$ in quality is achieved when compared with equal error protection (EEP). Figure 2 shows the performance of proposed UEP and EEP when source rate to transmit is less than transmission block target rate. At most, $3\text{-}\mathrm{dB}$ gains are achieved for proposed UEP over EEP. If all MGS enhancement NAL units have to be included in the transmission block (as in Refs. 6, 7) then the Eq. 7 changes slightly by setting $0\u2a7dK(i,j)\u2a7dN-1$ . It can be solved by starting from $K(i,j)=0$ and applying step 2 in proposed heuristic solution. Curves “UEP-all” in Fig. 2 show it is inferior to the proposed UEP scheme especially at high packet loss rate.

We take a subjective test of some results according to the double-stimulus continuous quality-scale method suggested by ITU-R BT.500-10.^{11} The mean opinion scores are rescaled to a range of 0–100. Difference mean opinion scores (DMOS) are calculated as the difference between the original video and the test video. Table 1 shows the DMOS of UEP and EEP at
$\mathrm{PLR}=30\%$
. It can be seen that subjective rating of the proposed method is better than the others.

## Table 1

DMOS comparison of the UEP scheme against EEP scheme at PLR=30%

Sequence L | EEP | Proposed | UEP all | Sequence L | EEP | Proposed |
---|---|---|---|---|---|---|

Harbour 1100 | 39.82 | 36.25 | 37.58 | Harbour 600 | 40.11 | 36.72 |

Mobile 1200 | 40.42 | 36.09 | 38.91 | Mobile 800 | 41.59 | 38.78 |

Foreman 600 | 39.13 | 30.74 | 34.02 | Foreman 200 | 39.98 | 33.79 |

Crew 1000 | 40.15 | 31.88 | 36.54 | Crew 500 | 40.51 | 36.15 |

## 4.

## Conclusions

We propose a UEP scheme by jointly selecting source packets and allocating unequal amounts of protection to MGS enhancement NAL units for every GOP. A heuristic algorithm is proposed to quickly get the solution. As illustrated by our simulation results, the proposed UEP scheme provides significant error resilience.

## Acknowledgments

This work was supported, in part, by NSFC (Grants No. 60502034 and No. 60625103), Hi-Tech Research and Development Program of China 863 (Grant No. 2006AA01Z124), Grant No. NCET-06-0409, and the 111 Project.

## References

**,” IEEE Trans. Circuits Syst. Video Technol., 17 (9), 1103 –1120 (2007). https://doi.org/10.1109/TCSVT.2007.905532 1051-8215 Google Scholar**

*Overview of the scalable video coding extension of the H.264/AVC standard***,” IEEE Trans. Circuits Syst. Video Technol., 17 (9), 1186 –1193 (2007). https://doi.org/10.1109/TCSVT.2007.906870 1051-8215 Google Scholar**

*Optimized rate distortion extraction with quality layers in the scalable extension of H.264 AVC***,” IEEE J. Sel. Areas Commun., 18 (6), 819 –828 (2000). https://doi.org/10.1109/49.848236 0733-8716 Google Scholar**

*Unequal loss protection: graceful degradation of image quality over packet erasure channels through forward error correction***,” IEEE Trans. Multimedia, 7 (4), 753 –765 (2005). https://doi.org/10.1109/TMM.2005.846782 1520-9210 Google Scholar**

*An unequal packet loss resilience scheme for video over the internet***,” EURASIP J. Appl. Signal Process., 1 –11 (2006). https://doi.org/10.1155/ASP/2006/45412 1110-8657 Google Scholar**

*Error-resilient unequal error protection of fine granularity scalable video bitstreams***,” IEEE Trans. Image Process., 15 (6), 1323 –1330 (2006). https://doi.org/10.1109/TIP.2005.864159 1057-7149 Google Scholar**

*GOP-based channel rate allocation using genetic algorithm for scalable video streaming over error-prone networks***,” IEEE Trans. Multimedia, 9 (1), 37 –45 (2007). https://doi.org/10.1109/TMM.2006.886339 1520-9210 Google Scholar**

*Two-dimensional channel coding scheme for MCTF-based scalable video coding***,” IEEE Trans. Consum. Electron., 54 (2), 736 –744 (2008). https://doi.org/10.1109/TCE.2008.4560155 0098-3063 Google Scholar**

*Layer-weighted unequal error protection for scalable video coding extension of H.264/AVC***,” 1 –6 (2007). Google Scholar**

*A novel adaptive unequal error protection method for scalable video over wireless networks***,” 352 –361 (2002). Google Scholar**

*Bit allocation in sub-linear time and the multiple-choice knapsack problem***,” (2000). Google Scholar**

*Methodology for the subjective assessment of the quality of television pictures*