1.

Introduction

The thriving interest of consumers for video content has brought the world closer to the universe of digital video provision than ever before. The recent market success of media services such as internet protocol television and video on demand through consumer electronic products, has created a considerable increase of the network traffic, which has caused the network operators to apply restriction policies, creating a controversial issue on the network neutrality and traffic differentiation.

This situation has created the need for more advanced encoding techniques (e.g., H.265/HEVC) and adaptation schemes¹ which achieve higher compression ratios and provide agility to the adaptation of the media service,² alleviating the network operators from the media-related network traffic. Thus, during the service provision, the video stream may be needed to be dynamically transcoded at different formats/profiles (e.g., such as in the paradigm of mobile-edge computing), resulting in adapted media services that dynamically fit the current network conditions and the terminal device specifications. However, this in-service transcoding process, which today can be supported by the emerging software-defined networking and network function virtualization techniques,³ introduces to the media service a wide variety of encoding impairments that degrade the deduced quality level of the encoded media service. Thus, the quality degradation of the media service, caused by the in-service transcoding process, creates the need for defining flexible video quality assessment (VQA) methods that will be able to evaluate the quality with the same accuracy as the well claimed video quality metrics [e.g., structural similarity (SSIM) index⁴]. These VQA methods would be able to assess the quality not only during the initial coding process of the source signal, but also across the media service delivery path to the end user, providing useful feedback both to the content provider for service adaptation actions and to the network operator for optimal traffic steering decisions.

To monitor the quality in real time and across the service provision chain,⁵ it is necessary to use flexible VQA tools, which are suitable for in-service integration, evaluating the media service along its network delivery path.

Currently, the available VQA methods are divided into two categories: the subjective and the objective ones. More specifically, the subjective VQA methods⁶ are based upon the opinion score of a group of viewers regarding the visual degradation of an encoded video sequence compared to the original uncompressed sequence, establishing them as the primary choice for video quality evaluation tests in terms of reliability, but also practically from a commercial perspective. The subjective video quality evaluation methods are expensive and time-consuming mainly due to their demanding setup within a controlled room/environment with sophisticated apparatus, which leads to the fact that they cannot be commercially exploited, especially within the service provision chain for monitoring purposes of the delivered video service.

Correspondingly, the objective VQA techniques are mathematical computational models that utilize various image characteristics (e.g., luma and chroma),^7–16 or other image statistics (e.g., blockiness),¹⁷ to approximate as well as possible the subjective test results in an efficient and cost-effective way. The objective methods are categorized into three groups determined by their approach and the metric used for the quality assessment: the full reference (FR) ones, the reduced reference (RR) ones, and the no-reference (NR) ones.

The FR methods evaluate the video quality by comparing the frames of the original video and the target video. The methods perform multiple channel decomposition of the video signal, where the objective method is applied on each channel, which feature a different weight factor according to the characteristics of the human visual system (HVS), using contrast sensitivity functions, channel decomposition, error normalization, weighting, and finally Minkowski error pooling for combining the error measurements into a single perceived quality estimation.¹⁸ Also, in the bibliography, FR methods for a single channel have been proposed where the proposed objective metric is applied on the video signal, without considering varying weight functions. Some FR metrics that are based on the video structural distortion have been proposed,¹⁹ among which is the widely known SSIM index, which has a very wide range of applicability across many different fields.^20–22 All FR methods, including SSIM, provide higher accuracy and credibility in comparison to the rest of the categories (RR and NR), but in the evaluation process, they require both the original and the encoded video sequences at the same site, making them inappropriate for integration in the service provision chain where the original signal is not available at the end-user site.

The RR methods are able to evaluate the video quality level based on metrics which use only some extracted structural features from the original signal.^23,24 The concept of the RR metrics was introduced by Refs. 7, 24, and 25, where the RR metric was based upon the extraction of various spatial and temporal features of the reference video, which are easily exposed to distortions added by the standard video compression process. RR metrics can be roughly categorized into three categories.

The first category includes all methods based upon models of low-level statistical properties of the original natural image. An RR metric that belongs to this category is described in Ref. 26, which provides a condensed amount of RR information obtained by the comparison of the marginal probability distribution of wavelet coefficients in different wavelet subbands with the probability density function of the wavelet coefficients of the decoded signal, while using the Kullback–Leiber divergence as a distance between distributions.

The second category of RR metrics includes methods that capture visual distortions, to quantify the decoded signal’s quality.^27–30 However, this type of metrics only performs well, when there is sufficient knowledge about the degradation process that the signal underwent. It is not efficient to apply these techniques to general cases whose distortion has not been previously assumed.

The third and the last categories of RR metrics are based upon models of the viewer’s perception, e.g., the HVS.^31–34 These models exploit and apply different psychological and psychological vision studies on the end users in an attempt to imitate the behavior of subjective test groups.

RR methods are more flexible for in-service integration since they require only partial information of the original video signal, but they have reduced accuracy and credibility in comparison to the FR metrics.

Finally, the NR methods evaluate the video quality on the basis of processing the frames of the target video alone. As they do not require any information from the original video sequence, they can be easily integrated within the service provision chain. However, their performance is limited to specific visual artifacts (e.g., tiling), restricting their range of applicability to special cases only. Thus, from the family of the objective methods, the RR and the NR metrics are more suitable for in-service integration, but they suffer from limited efficiency in comparison to the FRs, which offer high accuracy, but applicability limitations, since they require the original video sequence for assessing the video quality.

It is evident that RR methods require an additional communication channel within the network architecture to transmit the extracted features from the video provider site to the end-user terminals. It is obvious that the required bandwidth depends on the specific RR method. A bandwidth in the range of 1 to 150 kbps is usually required, depending on the RR method and the feature extraction type.^23,35 However, the method described in Ref. 35, which requires under 1-kbps bandwidth, performs very poorly in terms of absolute difference compared to subjective tests.

An alternative technique is that the extracted features (or in general the reference information) will be encapsulated inside the forward link, along with the video transmission. The more features that are extracted and transmitted to the end user, the more accurate is the objective video quality. However, more features require higher bandwidth of the communication network. So in RR methods, there is a trade-off between the accuracy of the VQA and the constraints in the network bandwidth.

In this paper, an RR method is proposed that is suitable for in-service use. The features extracted from both the original and the target video frames are based on the evaluation of the SSIM index. In this respect, the SSIM index is calculated for each frame of the original video using as reference a static white pattern, i.e., a video whose frames are all white. The SSIM index for each frame is transmitted to the end-user site. At the end-user terminal, the SSIM index is calculated between each frame of the received (target) video and the same static white pattern. The proposed RR metric is the ratio of the two SSIM indices. Through experimental measurements, it is shown that the proposed metric has a value very close to the SSIM index as calculated from the comparison of the original and the target video frames. The accuracy of the proposed metric as compared to SSIM depends on the video degradation, i.e., the higher the degradation, the worse is the accuracy. Experimental measurements show that for an acceptable video quality level,^36,37 the mean absolute percentage deviation (MAPD) of the proposed method is lower than 2.56%.

Another advantage of the proposed method is the low bit rate reference information signal that needs to be sent to the end user, which ranges between 400 and 600 bps.

The rest of the paper is organized as follows: Sec. 2 describes the use of SSIM as a feature extraction method from both the original and the target videos and introduces the metric SSIM RR (SRR), which can be directly compared to the original SSIM index. Section 3 presents a qualitative interpretation of the proposed video quality metric SRR. In Sec. 4, the performance evaluation of the proposed method is presented and analyzed, and finally Sec. 5 concludes the paper.

2.

Feature Extraction Using Structural Similarity

Among the most reliable objective evaluation metrics is the SSIM, which measures the SSIM between two image sequences, exploiting the general principle that the main function of the HVS is the extraction of structural information from the viewing field. If $x$ and $y$ are two video signals, then SSIM is defined as

In the typical SSIM index evaluation process, it is assumed that both the original and the target video sequences are available at the same site, as shown in Fig. 1. $\mathrm{SSIM}(x,y)$ evaluation for every frame can be based on any software implementation of Eq. (1), where $x$ is the original video sequence (${\mathrm{VS}}_{\mathrm{o}}$) in Fig. 1, $y$ is the target video sequence ${\mathrm{VS}}_{\mathrm{t}}$, and ${\mathrm{SSIM}}_{\mathrm{ot}}$ is their $\mathrm{SSIM}(x,y)$ index.

Fig. 1

Typical SSIM index evaluation with original and target video sequences available at the same site.

According to Ref. 4, where SSIM index is defined and introduced, SSIM comprises three image characteristics components: the luminance, the contrast, and the structure comparison. Their combination results in the widely used SSIM index. The three separate comparison components are

This motivated us to further research the sensitivity analysis of the SSIM accuracy under different $\alpha$, $\beta$, and $\gamma$ weights. More specifically, considering the purpose of this paper is to develop a flexible metric suitable for in-service applicability, we notice that the parameter $\gamma$ specifies the importance of the SSIM between signals $x$ and $y$, which is the most influential factor for the FR requirement in the applicability of the SSIM, since according to the following type, the ${\sigma }_{xy}$ factor needs to be calculated for the measurement of Eq. (5).

In the proposed method, in order to investigate the in-service applicability of the SSIM [i.e., in in-service cases where the calculation of Eq. (5) is not feasible due to the lack of the reference signal], we research in this paper the case that $\gamma \to 0$, so the relevant importance of Eq. (5) in the SSIM calculation is limited. Therefore, the requirement for the reference signal to be available together with the encoded signal for the estimation of Eq. (5) ceases to exist, allowing the decomposition of the SSIM index exclusively for the Eqs. (3) and (4) parameters, where no ex parameters exist that require the existence of both the reference and the encoded signal at the same place.

Based on this analysis, in the proposed method (see Fig. 2), the SSIM index is used as a tool to extract features from both the original and the target video sequences using a reference video pattern. In this respect, an initial ${\mathrm{SSIM}}_{\mathrm{or}}$ value is evaluated for every frame at the service provider site, by comparing the ${\mathrm{VS}}_{\mathrm{o}}$ with a video reference pattern (${\mathrm{VS}}_{\mathrm{r}}$), e.g., a video sequence of white video frames of the same resolution and frame rate, which is artificially generated [later in this section, the authors test the primary colors (i.e., white, black, red, green, and blue) as reference video patterns and it is shown that the white video pattern performs better than the others]. The evaluation of ${\mathrm{SSIM}}_{\mathrm{or}}$ can be based on any software implementation of Eq. (1), where $x$ is ${\mathrm{VS}}_{\mathrm{o}}$ and $y$ is ${\mathrm{VS}}_{\mathrm{r}}$ and refers to each frame. ${\mathrm{SSIM}}_{\mathrm{or}}$ can be considered as a feature of each frame of the original video and is sent to the end-user site by any means, i.e., either through the same communication channel as the video, or any other communication channel with a sufficient bandwidth, or by embedding it inside the transport stream of the video sequence. In any case, it is considered that the ${\mathrm{SSIM}}_{\mathrm{or}}$ value is recovered at the end-user site.

Fig. 2

SRR evaluation method using a reference video pattern as reference at both the service provider and end-user sites.

Referring to Fig. 2, an ${\mathrm{SSIM}}_{\mathrm{tr}}$ value is evaluated at the end-user site by comparing the received (target) video signal (${\mathrm{VS}}_{\mathrm{t}}$) with a reference video pattern (${\mathrm{VS}}_{\mathrm{r}}$), which is identical to the one used at the service provider site and is also artificially generated at the site. The evaluation of ${\mathrm{SSIM}}_{\mathrm{tr}}$ is based on the same software implementation of Eq. (1) which used at the transmitter site and refers to each frame of the target video sequence. ${\mathrm{SSIM}}_{\mathrm{tr}}$ can be considered as a feature of each frame of the target video.

The ratio of these two SSIM values can be considered as a new metric based on SSIM, namely SRR, i.e.,

Concerning the kind of reference video pattern, the applicability of the SRR method was evaluated based on using the primary colors (i.e., white, black, red, green, and blue) as reference video patterns. The selected primary colors portray a distinct RGB diversity, which is essential to demonstrate and evaluate the behavior of the proposed method, maintaining a simplicity in the implementation and representation of the pattern.

In this framework, the proposed method was applied on the experimental set of the 40 test signals, which are encoded with H-264 at three distinct quantization parameters (QP) values, specifically $\mathrm{QP}=12$, 22, and 32, each time utilizing a different primary color as a reference video pattern. The QP value regulates how much spatial detail is maintained during the encoding/compression process (modifying, respectively, the quantization step). A low QP value denotes a low quantization step, therefore, almost all the spatial detail of the video is retained, while a high QP value corresponds to high quantization step and the video spatial detail is aggregated resulting in increased distortion and degradation of video quality. For each color and QP value, the proposed SRR method was applied and each time the MAPD value in relevance to the SSIM was calculated.

The experimental results of the process are provided in Table 1, where it is observed that among the primary colors, the white video pattern provides a better performance across different QP values, while the rest of the primary colors perform notably worse than the white. Therefore, considering its better performance than the rest of the colors and its main characteristic as the easiest representation with only one luminance parameter, the white reference video pattern is recommended for implementing the proposed SRR method and it is selected for executing the experimental parts of this paper.

Table 1

MAPD for 40 test signals with different reference video patterns.

Concerning the channel requirements and the overhead of the proposed method, ${\mathrm{SSIM}}_{\mathrm{or}}$ is a number $<1$ and it can be represented by 2 bytes per frame for an accuracy of four decimal places (${10}^{-4}$). In this case, the required bit rate to be transmitted to the end-user site is 400 bps per $25\text{frames}/\mathrm{s}$. For an increased accuracy per frame of six decimal places (${10}^{-6}$), 3 bytes are required, resulting in 600 bps for the ${\mathrm{SSIM}}_{\mathrm{or}}$. Even 600 bps is significantly lower than the value of $\sim 1$ to 150 kbps required for other RR methods, as mentioned in Sec. 1.

3.

Qualitative Interpretation of the Proposed Video Quality Metric

In order to interpret the proposed quality metric and its relation to the SSIM index, we consider a video sequence which is encoded at various QP values, resulting in different bit rates and quality levels. The SSIM index between the original and the target video sequence for each frame is denoted as ${\mathrm{SSIM}}_{\mathrm{ot}}(\mathrm{QP})$, which is a descending function of QP. Given that the maximum value of ${\mathrm{SSIM}}_{\mathrm{ot}}$ is equal to 1, the SSIM variation is an exponential function as shown in Ref. 38. Therefore

The plot of ${\mathrm{SSIM}}_{\mathrm{ot}}(\mathrm{QP})$ versus QP is shown in Fig. 3, curve a. As QP increases, more and more information from the original video frame is lost, i.e., the spatial information (i.e., color and intensity of each pixel) of ${\mathrm{VS}}_{\mathrm{t}}$ is lost. In the extreme case, where the information is completely lost, all pixels are equal in color and density, i.e., the ${\mathrm{VS}}_{\mathrm{t}}$ is degraded to a sequence of uniform frames, as for example a white video pattern (similar to the used reference video pattern). In this extreme case, the lowest value of ${\mathrm{SSIM}}_{\mathrm{ot}}(\mathrm{QP})$ is the SSIM index between the original video sequence and the reference video pattern, which can be denoted as ${\mathrm{SSIM}}_{\mathrm{or}}$.

Fig. 3

Variation of: (a) ${\mathrm{SSIM}}_{\mathrm{ot}}$, (b) ideal ${\mathrm{SSIM}}_{\mathrm{tr}}$, and (c) real conditions ${\mathrm{SSIM}}_{\mathrm{tr}}$ versus QP.

The previous analysis refers to the comparison between the original and the target video frames. If we consider the comparison between the target video frames and the reference video (i.e., the white video pattern explained earlier), the ${\mathrm{SSIM}}_{\mathrm{tr}}(\mathrm{QP})$ index versus QP will be an ascending function.

This is because at low QPs the target frame is almost identical to the original and, therefore, will greatly differ from the (white) reference video pattern, resulting in a low ${\mathrm{SSIM}}_{\mathrm{tr}}$ value. Furthermore, as QP increases, the target frame will gradually become similar to the reference white video pattern, resulting in a higher value of ${\mathrm{SSIM}}_{\mathrm{tr}}$. For very high QP, the target frame is identical to the reference one and the maximum ${\mathrm{SSIM}}_{\mathrm{tr}}$ is equal to 1.

Considering an exponential variation of ${\mathrm{SSIM}}_{\mathrm{tr}}$ (similar to ${\mathrm{SSIM}}_{\mathrm{ot}}$), it is deduced that

From Eqs. (8) and (9), it can be deduced that

Comparing Eqs. (7) and (10), it is evident that

4.

Performance Evaluation

4.1.

Performance Comparison of Structural Similarity Reduced Reference to Structural Similarity

The performance of the proposed method is evaluated by comparing SRR with the original SSIM index (${\mathrm{SSIM}}_{\mathrm{ot}}$) for a large number of video frames. In this respect, a wide range of video sets were selected, which include 40 video sequences of various length, resolution, and content. The selected video sequences include 11 reference video sequences,³⁹ 2 long-duration sequences (Bigbuckbunny and Elephantsdream), and 27 nonreference video sequences retrieved from movie trailers. The total number of unique frames which were used for evaluating the proposed method is 60,866.

The original uncompressed video sequences were encoded at three QP values: 12, 22, and 32, which satisfactorily cover the achieved video quality range of the encoded/compressed video signals. Higher QP values were not examined because they lead to unacceptable video quality.³⁶ An initial qualitative comparison between SRR and SSIM index is shown in Fig. 4, which shows the variation of SRR and SSIM index for each frame of a video sequence (Kristen&Sara) with $\mathrm{QP}=32$. From Fig. 4, the qualitative similarity of RSS and SSIM index is obvious.

Fig. 4

Qualitative comparison of RSS and SSIM index applied on Kristen&Sara video sequence with $\mathrm{QP}=32$.

For the quantitative measurement of the performance of the proposed method, the MAPD for each frame $i$ between SRR and SSIM index is calculated. MAPD is a widely used metric for measurement of the accuracy of a prediction method, specifically in trend estimation such as the the proposed method

Eq. (12)

$$\mathrm{MAPD}=\frac{1}{n}\sum _{i=1}^n\frac{|{\mathrm{SSIM}}_i-\text{Predicted}\_{\mathrm{SSIM}}_i|}{{\mathrm{SSIM}}_i},$$

where ${\mathrm{SSIM}}_i$ is the SSIM index per frame $i$ and $\text{Predicted}\_{\mathrm{SSIM}}_t$ is the SRR value for the frame $t$, according to Eq. (7).

Table 2 presents the MAPD for the experimental set of the 40 video sequences at the three QP values (i.e., 12, 22, and 32) and the mean value of MAPD for each QP.

Table 2

MAPD for SRR and SSIM index for a white video reference pattern.

Test signal	Resolution	Frames	QP=12	QP=22	QP=32
Apocalypto1	$352\times 288$	990	0.003053	0.005581	0.033561
Apocalypto2	$352\times 288$	990	0.004793	0.005936	0.020171
Apocalypto3	$352\times 288$	990	0.005903	0.007334	0.021447
Apocalypto4	$352\times 288$	501	0.005136	0.009094	0.017331
Mission1	$352\times 288$	990	0.005578	0.007233	0.016888
Mission2	$352\times 288$	990	0.006228	0.006695	0.019181
Mission3	$352\times 288$	293	0.004423	0.005531	0.016902
Superman1	$352\times 288$	990	0.004448	0.006062	0.019283
Superman2	$352\times 288$	990	0.00459	0.0069	0.021657
Superman3	$352\times 288$	268	0.001328	0.003436	0.035995
Insideman1	$352\times 288$	990	0.003798	0.004191	0.019316
Insideman2	$352\times 288$	990	0.006988	0.008945	0.012651
Insideman3	$352\times 288$	990	0.00526	0.006288	0.012146
Insideman4	$352\times 288$	376	0.00125	0.001084	0.027495
Davinci1	$352\times 288$	990	0.0029	0.003661	0.014045
Davinci2	$352\times 288$	990	0.005846	0.007804	0.019533
Davinci3	$352\times 288$	990	0.005416	0.005909	0.012271
Davinci4	$352\times 288$	627	0.007838	0.009838	0.018827
Basic1	$352\times 288$	990	0.005603	0.006751	0.015259
Basic2	$352\times 288$	990	0.00652	0.009051	0.015006
Basic3	$352\times 288$	990	0.006734	0.00765	0.014208
Basic4	$352\times 288$	351	0.003842	0.004624	0.010905
16blocks1	$352\times 288$	990	0.004517	0.004992	0.017711
16block2	$352\times 288$	990	0.006962	0.006389	0.012574
16block3	$352\times 288$	990	0.005295	0.005044	0.01225
16block4	$352\times 288$	451	0.003401	0.00297	0.007796
Batman1	$352\times 288$	2659	0.010258	0.014053	0.042584
Batman2	$352\times 288$	913	0.00619	0.011052	0.070104
Bigbuckbunny	$640\times 360$	14315	0.01091	0.020527	0.039061
Elephantsdream	$640\times 360$	15691	0.008501	0.013748	0.040883
Basketballpass	$416\times 240$	501	0.011807	0.007371	0.022277
Bqsquare	$416\times 240$	601	0.006849	0.028219	0.101372
Bubbles	$416\times 240$	501	0.005385	0.015653	0.069205
Basketballdrill	$832\times 480$	501	0.016819	0.00973	0.0052
Bqmall	$832\times 480$	601	0.008256	0.005319	0.009498
Racehorses	$832\times 480$	300	0.009953	0.00794	0.022274
Partyscene	$832\times 480$	501	0.005572	0.01613	0.075519
Stockholm	$1280\times 720$	604	0.000514	0.030779	0.047909
Kristen&Sara	$1280\times 720$	600	0.008666	0.006363	0.010176
Foupeople	$1280\times 720$	600	0.009949	0.001049	0.001995
Mean value			0.00618198	0.00867315	0.02556165

Table 2 shows that the accuracy of the proposed method ranges from 0.62% ($\mathrm{QP}=12$) to 2.56% ($\mathrm{QP}=32$), which represents the worst case performance, showing that the proposed SRR method maintains a satisfactory performance across all the potential range of QP values, although better accuracy is achieved at lower QP values. The comparison between the correlation of QP and ground truth (i.e., MOS) versus the correlation of SRR scores and ground truth provides the result of 28.65, showing the advantage and the better performance of the proposed metric.

The experimental variation of MAPD versus QP is shown in Fig. 5(a), where the trend line is the dashed line. For comparison reasons, the theoretical MAPD is also shown in Fig. 5(b).

Fig. 5

Variation of MAPD versus QP: (a) experimental and (b) theoretical.

It is calculated from Eq. (12), where ${\mathrm{SSIM}}_i$ equals ${\mathrm{SSIM}}_{\mathrm{ot}}(\mathrm{QP})$, as calculated from Eq. (8). Also, $\text{Predicted}\_{\mathrm{SSIM}}_t$ equals SRR, i.e.,

Eq. (13)

$$\mathrm{SRR}=\frac{{\mathrm{SSIM}}_{\mathrm{or}}(\mathrm{QP})}{{\mathrm{SSIM}}_{\mathrm{tr}}(\mathrm{QP})},$$

where ${\mathrm{SSIM}}_{\mathrm{tr}}(\mathrm{QP})$ corresponds to curve c of Fig. 3, which refers to an exemplary practical case.

The slopes of the two lines of Figs. 5(a) and 5(b) are the same, which shows that the theoretically calculated SRR is very close to the experimental results.

4.2.

Performance Comparison of Structural Similarity Reduced Reference to Subjective Difference Mean Opinion Scores and Other Assessment Methods

According to the video quality experts group research,⁴⁰ in order to obtain a linear relationship between an objective assessment method score and its corresponding subjective score, each metric score $x$ is mapped to $q(x)$. The nonlinear best-fitting logistic function $q(x)$ is given as follows:

Eq. (14)

$$q(x)={\beta }_1\{\frac{1}{2}-\frac{1}{1+\exp [{\beta }_2(x-{\beta }_3)]}\}+{\beta }_4\chi +{\beta }_5.$$

The parameters (${\beta }_1$, ${\beta }_2$, ${\beta }_3$, ${\beta }_4$, and ${\beta }_5$) are calculated through minimizing the sum of squared differences among the subjective and the mapped scores. To compare the performance of a newly proposed SRR method with the existing ones, performance evaluation metrics are used, such as the Pearson’s linear correlation coefficient (LCC), which is the LCC between the predicted MOS and subjective MOS. LCC is a measure of prediction accuracy of an objective assessment metric, i.e., the capability of the metric to predict the subjective scores with low error. The LCC can be calculated using the below equation

Eq. (15)

$$\mathrm{LCC}=\frac{{\sum }_{i=1}^{M_d}(q_i-\overline{q})(s_i-\overline{s})}{{[{\sum }_{i=1}^{M_d}{(q_i-\overline{q})}^2]}^{\frac{1}{2}}{[{\sum }_{i=1}^{M_d}{(s_i-\overline{s})}^2]}^{\frac{1}{2}}},$$

where $s_i$ and $q_i$ are the subjective and the mapped scores for the $i$’th frame of a video of size $M_d$, respectively, and $\overline{s}$ and $\overline{q}$ are the means of the mapped and subjective scores, respectively. A good objective assessment metric is expected to have high LCC (close to 1) in contrast to MAPD, which should have low values (i.e., close to 0), as shown in Sec. 4.1.

Moreover, the Spearman rank order correlation coefficient (SROCC), which measures the monotonicity of the proposed method against subjective human scores, was also applied. SROCC is a nonparametric measure of statistical dependence between two variables, which assesses how well the relationship between two variables can be described using a monotonic function. If there are no repeated data values, a perfect Spearman correlation (equal to 1) occurs when each of the variables is a perfect monotone function of the other.

Therefore, in order to evaluate the performance of the SRR method and derive the LCC, following the aforementioned methodology, the Laboratory for Image and Video Engineering (LIVE) video quality dataset^6,41 was used. The LIVE video quality database (VQDB) uses 10 uncompressed high-quality videos with a wide variety of content as reference videos. A set of 150 distorted videos were created from these reference videos (15 distorted videos per reference) using H.264-based compression. Then each degraded video in the LIVE VQDB was assessed by 38 human subjects in a single stimulus study with hidden reference removal, where the subjects scored the video quality on a continuous quality scale. The mean and variance of the difference mean opinion scores (DMOS) were obtained from the subjective evaluations.

A scatter plot of proposed objective SRR scores versus DMOS for all H.264 videos in the LIVE VQDB is shown in Fig. 6 along with the best-fitting logistic function.

Fig. 6

Scatter plot of objective RSS scores versus DMOS with the best-fitting logistic function.

The SROCC and the LCC are computed between the objective and the subjective scores. Table 3 shows the performance of the proposed SRR model against other VQA methods, both FR and RR, in terms of the SROCC and LCC.

Table 3

Comparison of the performance of VQA algorithms—LCC and SROCC.

VQA method	Type	LCC	SROCC
RR-LHS42	RR	0.4557	0.4082
J.24633	RR	0.4488	0.4157
Peak signal-to-noise ratio (PSNR)	FR	0.5493	0.4585
Yang’s RR VQA43	RR	0.5654	0.5366
Visual signal-to-noise ratio (VSNR)44	FR	0.6216	0.6460
Proposed SRR method	RR	0.6260	0.5862
Video quality metric (VQM)34	FR	0.6459	0.6520
SSIM4	FR	0.6656	0.6514
RR metric35	RR	0.7567	0.7486

According to Table 3, the accuracy of the proposed SRR method is measured more accurately than the RR VQA methods RR-LHS,⁴² J.246,³³ and Yang’s RR VQA ⁴³ both in terms of LCC and monotonicity (SROCC), except for the RR metric,³⁵ which provides better results. Similarly, the accuracy of the proposed SRR method is better than those of the PSNR and VSNR FR VQA methods in terms of LCC, while it is slightly lower than the performance of VQM and SSIM index, as expected, due to the reduced reference nature of the proposed methodology. In terms of monotonicity (SROCC), the proposed method performs better than the PSNR, but lower than the rest of the VQA methods, without, however, significantly deviating from their performance range.

5.

Conclusions

This paper proposes an RR VQA method using SSIM index as a tool to extract features from both the original and the target video sequences, using a reference video pattern. The method is suitable for monitoring the video quality in real time and across the service provision chain. The performance of the proposed method was evaluated using a large experimental set of 40 reference and nonreference video sequences, with spatial resolution ranging from common intermediate format up to high definition, utilizing a static white video as a relative reference pattern. The proposed method maintains a satisfactory performance across all the potential range of QP values, although better accuracy is achieved at lower QP values. Moreover, comparison to subjectively evaluated scores of LIVE video quality dataset shows that the accuracy of the proposed method is better than the average performance of RR VQA methods and is within the performance range of the FR VQA methods. Finally, another advantage of the proposed method is the low bit rate reference information signal that must be sent to the end user, which ranges between 400 and 600 bps.