1.

Introduction

Many imaging sensors, including synthetic aperture radar (SAR), light detection and ranging (LiDAR) sensors, hyperspectral cameras, and wide-area motion imagery (WAMI) sensors, have been developed to obtain target scene images for various applications such as object detection, entity classification, multiple-target tracking, and activity-based intelligence.¹ The amount of the raw data obtained with these sensors is large. For example, a hyperspectral camera captures multiple (usually 100+) images of the same target scene with different wavelengths, and the amount of data obtained is usually in the order of several megapixels (an example is $200\times 200\times 115\text{pixels}$ in Ref. 2). Another example is WAMI, which generates images over city-sized areas to enable monitoring of vehicle and pedestrian movements.^3,4 A typical WAMI image data size is over 144 megapixels ($\mathrm{12,000}\times \mathrm{12,000}\text{pixels}$), and the next-generation WAMI image data size will be in the level of 1.6 gigapixels ($\mathrm{40,000}\times \mathrm{40,000}\text{pixels}$).⁵

To transmit the raw data to the users or processing units, either a wideband channel or a long-time interval is needed.^6,7 To reduce the required communication bandwidth or the transmission time, the raw data should be compressed. Lossless compression is able to preserve all the information, but has limited reduction power. On the other hand, lossy compression, which may result in very high compression ratio, suffers from interpretability loss as quantified by the National Imagery Interpretability Rating Scale (NIIRS).^8,9

NIIRS is a subjective quantification of image interpretability according to the types of tasks a certified image analyst (IA) is able to perform with the imagery for a given rating level. NIIRS has been defined for the following four types of imaging modalities: visible (EO), infrared (IR), radar (SAR), and multispectral.^10–13 NIIRS is a 10-level scale with each level defined by a set of information extraction tasks called criteria. Example criteria for EO, IR,^14,15 and SAR¹⁶ are given in Table 1. The criteria consist of a verb indicating the level of recognition (e.g., distinguish, detect, or identify), the target or object of interest (e.g., building), and some qualifier (e.g., type, size, or feature).

Table 1

Example EO, IR, RADAR, and multispectral NIIRS level 3 criteria for ground order of battle.

Imagery collection and the selection of the compression impact the performance of image fusion methods.¹⁷ When a user interprets an image as per high level information fusion,¹⁸ there is a need to understand the context in which the compression level is desired.¹⁹ Adaptive context management is desired to determine the correct level of performance desired.²⁰ One way to determine the balance between user needs and the compression level desired is through the performance analysis of image quality.^21–23 Through the use of image quality, various image processing methods have been developed for cloud architectures.^24,25 An open research question is the alignment of machine-level image interpretability with that of human observers,^26,27 although initial comparisons suggest the human perception and machine-level processing are sensitive to different image characteristics.^28,29 Many examples to compute the NIIRS have been reported¹¹ and updates are included in the Motion Imagery Standards Board. Recent efforts include the Video-National Imagery Interpretability Rating Scale (VNIIRS)^30–33 which can be used for video analysis,^34,35 but they still require extensive validation of how to be applied in dynamic imagery collections.

The general image quality equations (GIQEs) include GIQE3 (i.e., version 3), GIQE4, and GIQE5. GIEQ3 and GIQE4 were developed for hardcopy images.^36,37 GIQE5 focuses on softcopy images. Griffith presented a preliminary version of GIQE5.³⁸ Both hardcopy and softcopy methods are a function of ground sampling distance (GSD), relative edge response (RER), and the signal-to-noise ratio (SNR).

In this paper, we present the Compression Degradation Image Function Index (CoDIFI) framework that predicts the NIIRS degradation of an image due to compression. The foundation of this framework is the GIQE, which relates the image quality measure in NIIRS to sensor parameters and acquisition conditions with the goal of objectively predicting the NIIRS rating of images obtained from an imagery collection setting with known sensor parameters and acquisition setting. Specifically, parameters such as GSD, RER, edge overshoot ($H$), noise gain ($G$), and SNR are involved in GIQEs. Based on GIQE, we derive the general image quality degradation equations (GIQDEs) to predict the interpretability loss due to compression. A major feature of GIQDE is that it eliminates the inclusion of GSD, which cannot be inferred easily from the imagery, in its final form when the image quality degradation is a result of data compression.

In this paper, a two-stage CoDIFI framework is presented. In the first stage, automated image analytics estimate the RER as well as the edge overshoot of a given image. In the second stage, image analysis is performed on a synthetic binary edge image to build the CoDIFI model that relates NIIRS degradation and ratio of edge gradients before and after compression. Using the CoDIFI model, the compression-induced NIIRS degradation can be inferred by edge gradients obtained before and after compression. The proposed CoDIFI-NIIRS framework can be utilized to predict the NIIRS degradation for a given compression setting, thus, enabling a user to broker the maximum compression setting while maintaining a specified NIIRS rating.

This paper is organized as follows. In Sec. 2, two versions of GIQEs are reviewed. Section 3 derives the GIQDE for GIQE version 3. The automated image analytics developed to estimate edge profiles is presented in Sect. 4. In Sec. 5, the CoDIFI model construction is explained. Finally, experiments are presented in Sec. 6, as well as performance validation in Sec. 7 followed by conclusions in Sec. 8.

2.

General Image Quality Equation

The NIIRS rating of a given image is obtained from certified IAs, who are usually not available. Many efforts have been made to relate the measure of image quality in terms of NIIRS to sensor parameters, and the results are the GIQEs.³⁷ GIQEs predict NIIRS as a function of the imaging sensor and the acquisition setting of relevant parameters: GSD, RER, SNR, noise gain ($G$), and edge overshoot height ($H$). The parameters RER, $G$, and $H$ are defined after image enhancements are performed. The GIQE for an EO sensor is given as³⁷

GIQE3 was released in December 1994 to the unmanned aerial vehicle/sensors community whereas GIQE4 was published in November 1997 for the development of the commercial space imaging industry.³⁷ Both GIQEs were empirically determined using linear regression technique assuming hardcopy viewing. One major difference between GIQE3 and GIQE4 lies in the definition of GSD. In GIQE3, GSD is defined in the plane orthogonal to the line of sight while, in GIQE4, GSD is defined in the ground plane. The derivation and validation of GIQE4 was based on a set of 359 images whose characteristics are listed in Table 2.³⁸ Note that GIQE4 may not be accurate for an image whose characteristic is outside of the listed range. GSD is the actual ground distance in inches between two adjacent pixels. The GSD value is usually included in or has to be calculated from image metadata and cannot be obtained from simple image analysis.

Table 2

Characteristic range used to derive and validate GIQE4.

RER mainly affects the contrast of an image. It is estimated using the Stennis Space Center specified edge target as shown in Fig. 1(a) and its tilted version.³⁹ An RER value is obtained by estimating the slopes of edge profiles within the image as shown in Fig. 1(b). In principle, RER estimates effective slope of the imaging system’s edge response.

Fig. 1

(a) Stennis Space Center specified edge target for RER estimation. (b) Normalized edge response for RER estimation.

The edge overshoot height ($H$) of normalized edge response is the result of the application of MTFC, whose aim is to increase the image contrast but inevitably results in edge overshoot/edge ringing artifacts. Figure 2 shows the overshoot phenomenon, which is reproduced from Ref. 40. Figure 2(b) is obtained by applying small low-fidelity image-sharpening kernels on the image shown in Fig. 2(a). Edge-ringing artifacts are clearly observed in Fig. 2(b). The edge overshoot due to image sharpening is given in Fig. 2(c).

Fig. 2

An illustration of edge overshoot height due to image-sharpening. (a) Image before sharpening, (b) image after sharpening, and (c) edge overshoot height.

The application of MTFC inevitably amplifies noise. Noise gain $G$, due to the application of MTF, can be calculated from the coefficients, $w$, of MTFC kernels for a pixel $(m,n)$

For example, with the following MTFC, which is a symmetric $3\times 3$ sharpening kernel, the $G$ value can be computed to be 3.51.

Without the knowledge about the actual MTFC kernel, $G$ cannot be obtained. In Ref. 41, a fixed value of 4.16 was used for both Quickbird and IKONOS images. Fortunately, in the GIQDEs to be introduced next, the parameter $G$ is no longer involved when the change of SNR due to compression is small.

Finally, SNR is defined as the ratio between the power of image signal with the DC component excluded and the power of noise signal. For a given noise free image $I$, noise image $N$, and noise-corrupted image $Y=I+N$, the SNR is computed as

3.

Image Quality Degradation Equation

In an analysis of the general image quality equation,⁴² it is concluded that GIQE3 image quality predictions are more accurate than those from GIQE4 in a certain scenario. In addition, GIQE4 introduces discontinuity when RER is equal to 0.9. For this reason, we adopt GIQE3 instead of GIQE4 for the discussion in this section. However, similar discussion can be made using GIQE4.

Denote the GIQE estimated NIIRS for an image data before and after compression as

Note that, in both cases, the parameters GSD and $G$ are not changed as they are sensor setting related parameters and are not affected by compression. With Eqs. (5) and (6), the change of NIIRS due to compression can be derived as

By expressing the SNR after compression ${\mathrm{SNR}}_1$ as its Taylor series at ${\mathrm{SNR}}_0$, then

We call Eq. (8) the GIDQE, which predicts the interpretability loss due to compression. From Eq. (8), it is observed that the parameters GSD, $G$, and SNR involved in GIQE are no longer required to predict the interpretability loss due to compression.

4.

Image Analytics for Edge Profile Estimation

RER and edge overshoot height ($H$) are defined through edge profiles as can be seen in Figs. 1(b) and 2(c). This section presents an image analytic approach that performs edge profile extraction from which RER and $H$ can be estimated. Figure 3 shows the edge profile extraction workflow, which involves a number of modules such as Canny edge detector, Hough transform, and edge stripes determination.

Fig. 3

Edge profile extraction workflow along with a description of the output of each module and the corresponding sample output in Fig. 4.

The first two modules, Canny edge detector and Hough transform, are employed to extract line edges from the input image. For each extracted line edge, the edge stripes determination module extracts the corresponding edge stripe. A sample extracted edge stripe is provided in Fig. 4(c). Next, the edge intensity determination module is used to determine the intensity value that defines the edge point in each column of the edge stripes. Before an edge point can be decided, it is necessary to define the maximum and minimum intensity values, which will be normalized to one and zero, respectively. To this end, considering the possibility of having edge overshoot, the maximum intensity for an edge is taken as the median of the intensities of the first two rows of the bright side of each edge stripe. Rather than the minimum value, the minimum intensity is taken as the fifth percentile of the edge stripe to eliminate the possible outliers in each edge stripe.

Fig. 4

Illustration of the proposed edge profile extraction workflow. (a)–(g) are the outputs at various stages of the workflow depicted in Fig. 3.

Once the maximum and minimum intensity values are defined for each edge stripe, their mean value is taken as the intensity value that defines an edge point. After the edge intensity of each edge stripe is obtained, the edge center of each edge profile can be assessed. The edge profile is determined by searching for the location of each edge profile whose intensity is within a $\delta$-neighborhood of the edge intensity determined in the previous step. Here, $\delta$ is a predefined small value that determines the precision of the edge center, which is set it to be 0.001 in this work. Once the edge center is found for each edge profile, it is resampled at an array of positions centered at each edge center. Figures 4(d) and 4(e) show the edge profiles of an edge stripe before and after the edge profiles alignment module.

To compute RER value, edge profiles have to be normalized to from 0 to 1 as shown in Fig. 1(b). In the module Edge intensity determination, the maximum and minimum intensity values within an edge stripe have been decided. Denote them as $I_{\max }$ and $I_{\min }$. Edge profiles normalization is performed as $I_n=(I-I_{\min })/(I_{\max }-I_{\min })$, where $I_n$ is the normalized intensity and $I$ is the original intensity of a pixel of the raw edge profile, and $I_{\max }$ and $I_{\min }$ are the maximum and minimum intensity values determined for each edge stripe. Figure 4(f) shows the edge profiles normalized from those shown in Fig. 4(e). The last module, edge profiles finalization, produces a single edge profile for each edge stripe along with a quality measure. The final single edge profile is obtained by averaging the raw edge profiles of each edge stripe. However, not all averaged edge profiles of each edge stripe are equally reliable. The edge profiles module produces, in addition to the mean edge profile, a quality measure based on the variance of the raw edge profiles defined as

After obtaining the edge profile, it is straightforward to determine the RER by taking the difference of the edge profile values at location $+0.5$ and $-0.5$ as shown in Fig. 1(b). To estimate edge overshoot height, we follow the approach described in Ref. 43. First, obtain normalized edge profile from $-3$ to 3 pixels from edge center. Then, the maximum value between $+1$ and $+3\text{pixels}$ from the center is taken as the edge overshoot $H$ if it is greater than 1. Otherwise, the value at $+1.25\text{pixel}$ from the edge center is used as $H$ value. This approach is graphically shown in Fig. 5. There are two cases in Fig. 5. Case 1 indicates undershoot in the edge profile. In this case, the value at position 1.25 pixel is adopted as $H$ value, which is about 0.8. In case 2, the maximum value between 1 and 3 occurs at position 1.75 with value 1.2, which is greater than 1. Therefore, the $H$ value in case 2 is determined to be 1.2.

Fig.5

Edge overshoot height estimation from edge profile.

5.

Compression Degradation Image Function Index Model

As will be seen in the experimental results presented in Sec. 6, the estimated NIIRS degradation is not a smooth function of compression ratio by directly plugging the parameters estimated in Sec. 4 in Eq. (8). This is not surprising as the image analytics presented in Sec. 4 is able to reliably extract edge profiles only in the simplest case, for example, a binary edge image as displayed in Fig. 6. Therefore, for images that lack clear, strong, straight line edges, the proposed edge profile extraction approach is doomed to failure. For this reason, an alternative approach is desired for practical application.

Fig. 6

An ideal edge for edge profile extraction.

From the derived GIQDE given by Eq. (8) and the definitions of RER and $H$ as shown in Figs. 1 and 2, it seems reasonable to conclude that NIIRS degradation is directly related to the gradients calculated at edges before and after compression. For this reason, we assume that NIIRS degradation can be modeled as a function of the ratio of edge gradients obtained before and after compression. That is,

Once this model $\pi (·)$ is available, reduction of NIIRS rating can be predicted simply by the ratio of gradients obtained before and after compression at the same edge points. In this work, a simple neural network (NN) (shown in Fig. 7) is employed to obtain the $\pi (·)$ model.

Fig. 7

Structure of the NN employed to model the relationship between the gradient ratio and NIIRS degradation.

The training data are obtained by applying the image analytics presented in Sec. 4 to a series of sequentially degraded images of the image given in Fig. 6. The degraded images are generated by sequentially blurring the simulated edge image shown in Fig. 6 with fixed sized ($[23\times 23]$) Gaussian low-pass filters with different standard deviation values ranging from 0.2 to 3. Figure 8 shows some of the resulting degraded images. After the generation of training data, a simple NN with one hidden unit was trained to model the relationship between the gradient ratio and NIIRS degradation. Figure 9 shows the resulting CoDIFI where the blue asterisk symbol indicates the training data.

Fig. 8

Sample degraded images used to generate the training data. Degraded edge image (a) std = 0.2, (b) std = 2, and std = 3.

Fig. 9

The obtained CoDIFI based on GIQE version 3.

6.

Experimental Comparisons of Estimated NIIRS Loss

In the experiments, two approaches are used to estimate NIIRS degradation due to compression. The first approach directly applies Eq. (8) with parameters estimated from the automated image analytics presented in Sec. 4. The second approach applies the CoDIFI model constructed in Sec. 5 to estimate NIIRS degradation due to compression. A more detailed description follows.

Approach 1: Estimation of RER and $H$.

Approach 2: gradient ratio at edge points

Ten urban and 10 rural images as shown in Figs. 10 and 11 are used in the experiment. Urban images are characterized by more distinctive edges than rural images; thus, the edge profile-based approach is expected to work more reasonably in urban images. For compression schemes, JPEG and JPEG2000 are adopted and experimental results are shown in Figs. 12Fig. 13Fig. 14–15.

Fig. 10

Ten urban images used in the experiment. (a)Urban image 1, (b) urban image 2, (c) urban image 3, (d) urban image 4, (e) urban image 5, (f) urban image 6, (g) urban image 7, (h) urban image 8, (i) urban image 9, (j) urban image 10.

Fig. 11

Ten rural images used in the experiment. (a) Rural image 1, (b) rural image 2, (c) rural image 3, (d) rural image 4, (e) rural image 5, (f) rural image 6, (g) rural image 7, (h) rural image 8, (i) rural image 9, and (j) rural image 10

Fig. 12

Experimental result using JPEG compression on urban images.

Fig. 13

Experimental result using JPEG2000 compression on urban images.

Fig. 14

Experimental result using JPEG compression on rural images.

Fig. 15

Experimental result using JPEG2000 compression on rural images.

Figure 12 shows the results for the JPEG compression and Fig. 13 for JPEG2000 using the urban images.

In Figs. 12–15, the NIIRS degradation is plotted as a function of compression ratio and is estimated by the edge profile-based (EPB) approach and the CoDIFI model. From these two approaches over two scenarios, the following observations are presented to provide a high-level assessment of comparisons.

From these two approaches, the following observations resulted.

The first two observations reveal the limit of the EPB approach. That is, EPB performs well only when the line edges can be reasonably extracted. Since many line edges can be observed in urban images, the results from urban images are more reasonable than those from rural images, where line edges are rarely present. They also imply the validness of the CoDIFI-based approach because it produces similar but much smoother curves. The third observation indicates that JPEG compression may outperform JPEG2000 in terms of interpretability loss at very low compression rate. The roughness of the curves resulting from applying JPEG compression on rural images may due to the blocky artifact of JPEG compression as well as the lack of line edges in rural images.

7.

Validation of the CoDIFI Method

Experimentation using an independent NIIRS-assessment method provides an empirical validation of CoDIFI. Previous research has demonstrated the value of NIIRS-based methods for assessing compression of imagery⁹ and video data.^45,8,46 Additional investigations have shown relationships between loss in image interpretability and objective image metrics.³⁵ The approach is to compare the NIIRS loss as rated by expert human observers to predict NIIRS loss reported by CoDIFI.

The imagery data used for validation are new images that were not used in the development of CoDIFI. A set of still frames were extracted from Air Force Research Laboratory’s VIVID data set available from the Sensor Data Management System.⁴⁷ The VIVID data are described in Ref. 48 with a video analysis in Ref. 49. The validation set consisted of 30 images extracted from the VIVID data set (Fig. 15). Images ranged from NIIRS 4 to almost NIIRS 7 and included a range of scene content, backgrounds, and viewing geometries. For a given compression method, each of these “parent” images were compressed at various compression ratios to produce multiple compression products. Ratings by four expert human observers quantified the delta NIIRS between the parent image and each compressed image. Likewise, CoDIFI generates a predicted NIIRS loss for each compressed image. The comparison of these two ratings for the images in Fig. 16 is the basis for the validation analysis.

Fig. 16

Sample images used for the validation experiment.

Analysis using H.264 compression will demonstrate the validation process. The approach is iterative in which validation analysis supports improvements in CoDIFI and subsequent analysis confirms the improvement in performance. The relationship between the CoDIFI-predicted NIIRS loss and the expert observer delta-NIIRS ratings demonstrates that the two values align well. A slope of 1 and an intercept of 0 would indicate perfect alignment between the CoDIFI predictions and the expert ratings. The regression analysis yields a slope of 0.972, which is not statistically different from 1 ($t\text{-}\text{statistic}=0.718$). The intercept value of 0.15, which is statistically different from zero, is small in practical terms. Visual inspection of the data confirms the strong agreement shown by the regression model (Fig. 17) (Table 3).

Fig. 17

Comparison of CoDIFI predicted NIIRS loss and the expert observer delta-NIIRS ratings.

Table 3

Regression analysis (R2=0.94).

8.

Conclusion

In this paper, we presented the CoDIFI-NIIRS framework that can be employed to predict the compression-induced image quality loss in terms of NIIRS. The CoDIFI model is built on an automated image analytic that estimates line edge profiles on simulated edge images. The EPB approach in turn estimates the NIIRS degradation based on the derived GIDQE. Though our CoDIFI framework produces reasonable results, it is not fully validated. Our validation analysis demonstrates that CoDIFI predictions of NIIRS loss align well with expert ratings from human observers.

Future efforts include assessing the timeliness of the methods for automated systems, extensions with video sequences subject to motion artifacts, blur,⁵⁰ and resolution changes, and applicability to image fusion.⁵¹