Due to the launch cost and technical limitations, most Earth observation satellites, such as IKONOS, QuickBird, GeoEye, WorldView-2, GaoFen-1, and GaoFen-2, provided a high-resolution (HR) panchromatic (PAN) image and a low-resolution (LR) multispectral (MS) image with several spectral bands.12.3.–4 With the increasing requirement of higher spatial and spectral resolution remote data for various applications, such as feature extraction, land-cover classification, and climate change evaluation, the ability to capture high-quality images with lower cost is becoming increasingly more important. Pansharpening is a special data fusion, which emerged as the practical requirement. It integrates the complementary spectral and spatial characteristics from the provided MS image and the PAN image into the desired product.2 Up to now, a large number of pansharpening approaches have been developed. Broadly, three methodologies have been commonly used, namely, the component substitution, the high-frequency information injection, and the model-based methods.
The basic framework of component substitution methods is to transform the MS image into other space using a suitable transformation, and then the intensity channel or the first principal component is substituted by the PAN image. The classical component substitution methods are the intensity–hue–saturation (IHS)4 and principal component substitution.5 These methods may yield poor results in terms of spectral fidelity.6 In addition, due to an inadequate spectral model of the pansharpening method, spectral distortion is also generated. Some variant improvements to the original IHS-based method have been proposed. Rahmani et al.7 proposed an adaptive IHS (AIHS) method, which tried to represent the intensity component by an adaptive linear combination of the MS bands with the combination coefficients obtained by solving an optimization problem. Masoudi and Kabiri8 proposed a new IHS method using texture analysis and genetic algorithm adaption.
The high-frequency information injection methods can be briefly summarized as the sequential operations of details extraction and injection.910.11.–12 Since most of the directional and structural information is contained in the PAN image, several researchers proposed using wavelets11 and contourlet transforms12 to capture it from the PAN image. Then, the details missed in the LR MS image can be extracted and supplemented from the PAN image. Compared with the component substitution methods, although this method preserved better spectral information, the spatial distortions may occur accompanied with some blurring and artifacts.
The model-based fusion approach is another important category. On the basis of the image restoration, some research regards the solution of the fused image as an inverse optimization problem.1314.15.16.–17 Recently, inspired by sparse representation techniques, some researchers have achieved great success in data fusion.1819.20.–21 The initial work is proposed by Li and Yang.18 Then, Jiang et al.19 proposed a two-step sparse coding method with patch normalization (PN-TSSC). Zhu and Bamler20 proposed a new pansharpening method named sparse fusion of images (SparseFI), which explores the sparse representation of MS image patches in a dictionary trained only from the PAN image. To take into account the signal correlation among individual MS channels, the jointly SparseFI (J-SparseFI) algorithm was proposed.21 Although these methods perform well, the sparse coefficients of LR MS image patches are assumed to be identical to the corresponding HR MS patches. Due to complex and variant image content in the real-world images, the mappings between sparse coefficients of LR MS and that of HR MS should be complex.22,23
To address this issue, we propose a pansharpening approach via sparse regression aimed at finding the intrinsic and implicit relationship among the sparse coefficients to improve the robustness and stability of the remote sensing image pansharpening. To achieve this goal, on the one hand, we learn a pair of compact dictionaries relied on the sampled patch pairs from the high- and low-resolution images. The learned dictionary pair can greatly characterize the structural information of the LR MS and HR MS images. On the other hand, taken the complex relationship between the coding coefficients of LR MS and HR MS images into account, we model the complex relationship by a ridge regression and an elastic-net regression. The ridge regression characterizes the relationship of intrapatches. The elastic-net regression describes the relationship of interpatches. The theoretical analyses and experimental results in this paper indicate that the proposed method can generate competitive fusion results. The flowchart of the proposed method is summarized in Fig. 1.
Pansharpening via Mapping of Sparse Coefficients
In this section, the scheme of dictionary learning and the ridge regression of intrapatches are introduced at first. Thereafter, the interpatches regression mapping based on elastic-net model is discussed in detail. Finally, we present the high-resolution MS images reconstruction.
Dictionary Learning and the Ridge Regression of Intrapatches
The HR PAN image patches and the corresponding LR PAN image patches have the sparse coefficients and under the corresponding dictionaries and , respectively. Inspired by Wang et al.,22 we make an assumption that there is an implicit mapping function between the sparse coefficients of LR MS and HR MS patch pairs. In addition, the sparse coefficients of LR PAN and HR PAN patch pairs share the mapping function (see Fig. 2), which can be modeled by the following linear ridge equation:
However, OLS often does poorly in prediction. Thus, corresponding penalization techniques have been proposed to improve OLS to get a particular solution with desirable properties. For example, ridge regression24 and lasso25 are two popular and representative methods.
In our work, to get the mapping function with low computational burden and stable least square solution, we impose an -norm regularization penalty. The regularization term is included in the above-mentioned minimization
To enforce that the image patch pairs have the corresponding sparse coefficients with respect to HR dictionary and LR dictionary , a joint learning model below is proposed to find the desired dictionary pair as well as the desired intrapatches mapping
Given , , , , , and , Eq. (4) can be rewritten as follows:26,27 Thus, initialized the mapping function as an identity matrix and the dictionary pairs by DCT basis, the optimization performs in an alternative scheme over three stages. We call them “sparse coefficients update,” “dictionary update,” and “mapping function update,” which are corresponding to Eqs. (6)–(11). In other words, Eq. (5) is solved by translating to three subproblems.
In the sparse coefficients update stage, the mapping function and dictionary are fixed, and we can get the sparse coefficients and as follows:28
Similar to the sparse coefficients update stage, with the sparse coefficients fixed, we update the dictionaries according to the following equations:29 can be used to solve .
At last, we update the mapping function fixed the dictionaries and sparse coefficients
Learning Interpatch Regression Mapping Based on Elastic-Net Model
The above-mentioned regression mapping learning model just express the relationship of the sparse coefficients of image intrapatches. In Sec. 2.2, the relationship between sparse coefficients and all of the sparse coefficients of LR patches is taken into account.
Let be the predictors and be the response.30 Then, the model between response and predictors can be assumed asFig. 3. The aim of this relationship is to seek the weight vectors .
Given ;, the response is centered, and the predictors are standardized. Then, we reformulate Eq. (13) as
Based on elastic-net regression model,30 we proposed the following model:
Let us define , , , and . Then, the elastic-net model can be described as a lasso-type problem
The forward–backward operator splitting algorithm31 is employed to solve the problem32
High-Resolution Multispectral Image Reconstruction
After learned the dictionary-pairs , , the intrapatch mapping function and interpatches mapping function from the HR PAN image patches and the degraded ones, we can reconstruct the HR MS image under the following assumptions:
1. The input LR MS images and LR PAN image share the LR dictionary .
2. The reconstructed HR MS images and HR PAN image share the HR dictionary .
3. The sparse coefficients of LR MS images and HR MS images share the learned mapping function and .
The sparse coefficients of the ’th band of the LR MS image can be calculated by following step (each band is processed independently):
Then, we generate the corresponding coding coefficients associate with the regression function and , respectively
Finally, each band can be reconstructed as
Experimental Results and Analysis
To assess the performance of the proposed method, both simulated experiments and real data experiments are carried out. The simulation experiments are based on the strategy proposed by Wald et al.33 First, the original PAN and MS images are blurred with a low-pass filter and downsampled by a decimation factor 4 to obtain a degraded PAN image and MS images. Then, these degraded PAN and MS images are used to yield the HR MS images with the same spatial resolution to the original MS images. Finally, the fused HR MS images are compared with the original MS images. The QuickBird and WorldView-2 data are employed to test the performance of the proposed method. Five typical evaluation metrics are adopted to quantitatively evaluate the pansharpened results. The correlation coefficient (CC)34 and root-mean-square error (RMSE) are calculated for each band between the fused MS images and the reference original MS image. Erreur relative globale adimensionnelle de synthèse (ERGAS)33 and ,35 which are two comprehensive evaluation indexes, provide unique measures of the fusion performance for all the MS bands. Furthermore, the spectral angle mapper (SAM) index34 is also considered to measure the spectral distortion. Smaller values of RMSE, SAM, and ERGAS tend to be achieved by a better fusion result, as do larger CC, and Q4 values. In the real data experiments, since there is no reference HR MS image, the “quality with no reference” (QNR) measurement is used to evaluate different pansharpened results objectively.35
The proposed method is compared with five popular fusion algorithms: AIHS,7 Wavelet,11 PN-TSSC,19 SparseFI,20 and J-SparseFI.21 The results of AIHS method are gotten from the software developed by Rahmani et.al. The implementation of the AIHS method is available online in Ref. 36. The default parameters given in their implementations are adopted. Two levels of decompositions are used for the Wavelet method. The LR patch size in the SparseFI method is . A total of 10,000 patch pairs are selected to construct the dictionary pairs. As to J-SparseFI method, patch size is , and the dictionary size is 1000.
For our proposed pansharpening method, there are several parameters to be selected. In our experiments, we set the weight parameter , the regularization parameters , , , and , , the patch size , and the dictionary size 512. In Sec. 3.1, we provided a recipe about how to select these parameters to achieve a promising pansharpened result. Experimental results using parameters selected according to this recipe will be presented through the evolution curves and final fused images on different datasets.
In this section, we investigate the effect of the weight parameter and the regularization parameters , , , and , of the proposed method.
1. The effect of the weight parameter : The pansharpened performance obtained by the proposed method has strong links with the regularization parameter . To demonstrate it, we present the influence of the proposed method by choosing different weight parameters (). CC, SAM, and Q4 are evaluated to determine . The corresponding curved lines are plotted in Figs. 4 and 5, respectively. As shown in Fig. 4, when , which can be regarded as the case of ridge regression of intrapatches, the performance of the proposed method is restricted. With the increase of , more benefits on performance can be gained. This implies that the interpatches regression model is also essential for the fused result. However, we should also see that the value could not be set too high. Therefore, with a proper regularization parameter , the proposed method will gain the best results. A similar conclusion can be drawn for the QuickBird data in Fig. 5. Among the above resulted images, quantitatively, the best pansharpening result is yielded at .
2. The influence of the regularization parameters , , , and , : In this section, we probe how the regularization parameters affect the pansharpened performance of the proposed method. To test the influence, we tune one of them with the other fixed. Tables 1 and 2 are the objective assessment results of the proposed method on WorldView-2 data according to different values of and . Table 3 reports the influence of on QuickBird data. But it should reinforce the point that the pansharpened results are consistent with different test datasets under different parameters. From the tables, we can see that the regularization parameters play important roles in the final pansharpened performance. Little worse performance will be obtained when small deviations of the regularization parameters are around the selected values. If one tunes them far away from the value set by us, some worse performance will appear in the final fused image. When , , and are in the order of 0.01, 0.01, and 0.3, respectively, the proposed method can continually obtain a stable and optimal performance. The same thing happened again to other regularization parameters , , and . For balancing the reconstruction error, we set , , , , , and .
The effects of the proposed method under different λ1 on WorldView-2 data.
The bold and italic values represent the best results.
The effects of the proposed method under different λ2 on WorldView-2 data.
The bold and italic values represent the best results.
The effects of the proposed method under different γ1 on QuickBird data.
The bold values represent the best results.
Effects of Patch Size
As described in Sec. 2, our proposed method is based on patches learning. Thus, the effects of patch size on the pansharpening performance are evaluated in this section. To be fair, the dictionary size is fixed as 512. Four different patch sizes for LR PAN image are studied, including , , , and . Figure 6 shows how the patch size affects the visual quality of the fused results on WorldView-2 data. We can observe that the difference of spectral distortion is very small under different patch sizes. To illustrate well the performance change with the patches sizes, the quality indexes are calculated, where the average CC and RMSE of eight bands are presented. In addition, all the values of indexes are normalized to the range [0, 1]. The normalized results with respect to the different patch size are plotted in Fig. 7, where the horizontal axis is the patch size, and the vertical axis is the normalized results. Larger CC and Q4 indicate better fused result, and smaller RMSE, SAM, and ERGAS mean better result. Based on the curves in Fig. 7, it can be seen that when the patch size is , the best performance of the proposed method is obtained. The performance of patch size is slightly worse. However, the proposed method has less space complexity at this time. Taken into account a trade-off between the practical application in the future and the performance, we choose the patch size in the following experiments.
Effects of Dictionary Size
The above experimental results, we fix the dictionary size to be 512. In general, larger dictionaries should possess more expressive power and thus may yield more accurate approximation while increasing the computation cost. In this section, we evaluate the effect of dictionary size on pansharpening, i.e., the number of atoms in dictionary. From the sampled image patch pairs, we train four dictionaries of size 256, 512, 1024, and 2048 and apply them to the same remote sensing image. The results are evaluated both visually and quantitatively in Figs. 8Fig. 9–10 and Table 4.
The objective indexes of the pansharpened images with dictionaries of different sizes.
The bold values represent the best results.
Figure 8 shows the fused results for the QuickBird image using dictionaries of different sizes. Human visual system is not sensitive to the weak spectral distortions. Always we justify the distortion from the color change.10 Thus, in Fig. 9, we display the difference of pixel values measured between each pansharpened image and the reference HR MS image. Deep blue represents the smallest difference, whereas the red means the largest difference. Figure 9 shows the difference image under different dictionary sizes. While there are not many visual differences for the results using different dictionary sizes from 256 to 2048, we indeed observe the artifacts will gradually diminish with larger dictionaries (i.e., the subtle differences in the yellow circle in Fig. 9). In Table 4, we list five indexes of the pansharpened image for dictionaries of different sizes. As shown in the table, using larger dictionaries will yield better quantitative indexes. However, the computation is approximately linear to the size of the dictionary, i.e., larger dictionaries will result in heavier computation. Figure 10 shows the computation time in seconds with the fused image. In practice, one chooses an appropriate dictionary size as a trade-off between pansharpening quality and computation cost. We find that dictionary size 512 can yield decent outputs while allowing fast computation.
Simulation Results and Analysis
The size of the LR MS image in the simulated WorldView-2 data experiment is with eight bands and the corresponding PAN image sized . In Fig. 11(i), there are many varieties of ground objects, such as vegetation, buildings, and roads. In the aspect of the visual effects, the Wavelet method suffers from both spectral distortion and blocky artifacts in the building regions, as shown in Fig. 11(d). Meanwhile, the proposed method and J-SparseFI method show especially significant improvement in the spatial resolution. The details in the final fused images are as clear as that in the PAN image. AIHS and PN-TSSC methods are not good enough at improving the spatial resolutions. In Figs. 11(c) and 11(e), a lot of details are lost in the buildings, and the noise is introduced into the fused images. Compared with AIHS and PN-TSSC methods, SparseFI method is good in injecting more details. However, there are phenomena of the spectral distortions. The color of the whole fused images is dark. The visual effects are not close to the reference HR MS image.
To evaluate the performance of various methods objectively, Table 5 presents the objective evaluations about different methods. On the whole, the proposed method demonstrates the best objective performance, i.e., ranking as the first for the CC, RMSE, SAM, ERGAS, and Q4. The CC value of J-SparseFI method is inferior to the proposed method. It means J-SparseFI method cannot preserve the spectral information as well as the latter. The high Q4 value represents its effectiveness of improvement of the spatial resolution. The low SAM and ERGAS values of AIHS, PN-TSSC, and SparseFI methods show their shortage of spectral information of preservation. We also noticed the values of Q4 are very high, which reflect that the SparseFI and J-SparseFI method have the advantage of improving the spatial resolution.
Objective performance for different pansharpening methods on WorldView-2 data.
Figure 12 shows the difference of pixel values measured between each pansharpened image and the reference HR MS image. We can see that all the fused images by different methods have a lot of areas with blue and deep blue from the corresponding difference images. Fortunately, the proposed method shows the best performance. The least area of the red part in Fig. 12(f) indicates that the proposed method has a small number of outliers, which is consistent with the above objective and subjective analysis.
Experiments on Real Data
To verify the effectiveness of the proposed method in practical applications, we move on to conduct the experiments on the real data. Figures 13 and 14 present the fused results using six different methods. In Fig. 13, we can see that the AIHS method has the problem of textures overenhancement, as shown in the mountain. Figures 13(e)–13(g) are the pansharpened images provided by the PN-TSSC, SparseFI, and J-SparseFI, respectively, which can obtain promising results without causing obvious spectral and spatial distortion. Since our work illustrates the complex mapping relationship between the LR MS and HR MS image, experiments are also performed on complex urban scenes, as shown in Fig. 14. The final pansharpened results of various methods are presented in Fig. 14(c)–14(h). By the comparisons of all results, the component substitution method (AIHS) and the model-based methods (PN-TSSC, SparseFI, J-SparseFI, and proposed) outperform the high-frequency information injection method (Wavelet) in preserving the spectral information. Specifically, the proposed method produces the natural and satisfactory pansharpened images, which has similar spatial structures with the PAN image and similar spectral information with the MS image, as shown in Fig. 14(h). Table 6 and 7 show the quantitative assessment results.35 The ranking of QNR scores can confirm that the proposed method is better than the other methods in sharpening the real data.
Comparison of the proposed method with other methods on real data shown in Fig. 13.
Comparison of the proposed method with other methods on real data shown in Fig. 14.
All the methods are implemented in MATLAB® 2010b and run on an Intel Core firstname.lastname@example.org-GHz PC with 32-GB RAM. For fusing a PAN image and MS images, the AIHS and Wavelet methods need less than 1 s. The PN-TSSC, SparseFI, and J-SparseFI methods take about 147 s, 115 s, and 5 min, respectively. The running time of our proposed method is 133 s. Compared with these methods, the running time of the patch-based methods is still room for improvement. However, it is also reasonable to believe that with the rapid development in computer hardware and computation techniques, the time cost of the proposed method will soon no longer be an issue. We also can learn numerous projection matrices from the LR MS feature spaces to the corresponding HR MS feature spaces, which may decrease the computational time in the pansharpening phase. This will be investigated in our future work.
In this paper, combining ridge regression and elastic net of sparse representation, a pansharpening method is proposed for merging a PAN image with an MS image. The method used HR PAN image patches and their degraded ones to construct a training database. Then, the semicoupled dictionary learning method is used to train the LR–HR dictionary pair, which is to depict the MS structural information. The relationship between coding coefficients of LR MS and HR MS image patches is predicted by the within-patch ridge regression and among-patch elastic-net model. Our method can enhance the spatial resolution of MS images while reducing the distortion of spectral. The experimental results demonstrate that the proposed method can be compared with other state-of-the-art fusion methods.
This work was supported in part by the Fundamental Research Funds for the Central Universities (Grant No. LGZD201702), the National Natural Science Foundation of China (Grant Nos. 61302178, 61171165, 11431015, and 61571230), the Foundation of Guangxi Province (Grant No. 2014GXNSFAA118360), the Foundation of Jiangsu Province (Grant No. BK20161055), and the National Scientific Equipment Developing Project of China under Grant No. 2012YQ050250.
Songze Tang is currently an assistant professor in the Department of Criminal Science and Technology, Nanjing Forest Police College. He received his BS degree in information and computation science from Anhui Agriculture University, Hefei, China, in 2009, and his PhD in computer science and technology from Nanjing University of Science and Technology, Nanjing, in 2015. His research interests include computer vision and biometrics. He serves as a reviewer for three international academic journals.
Liang Xiao is currently a professor at the School of Computer Science, Nanjing University of Science and Technology (NUST). He received his PhD in computer science from NUST, China, in 2004. He has authored two books and around 100 technical articles in refereed journals, including the IEEE Transactions on Image Processing, the IEEE Transactions on Geoscience and Remote Sensing, and Pattern Recognition. His main research areas include computer vision and pattern recognition.
Pengfei Liu is currently an assistant professor at the School of Computer Science, Nanjing University of Posts and Telecommunications. He received his BS degree in information and computation science from Hefei Normal University, Hefei, China, in 2011, and his PhD in computer science and technology from Nanjing University of Science and Technology, Nanjing, in 2016. His research interests include computer vision and remote sensing data processing.
Lili Huang is currently an associate professor at Guangxi University of Science and Technology, China. She received her PhD in pattern recognition and intelligent systems from Nanjing University of Science and Technology, Nanjing, Jiangsu, China, in 2012. Her research interests cover image processing, image modeling, and superresolution reconstruction.
Yang Xu received his PhD in computer science from the Nanjing University of Science and Technology (NUST), Nanjing, China, in 2016. Currently, he is working as an assistant professor in Nanjing University of Science and Technology. His research interests are in the area of hyperspectral image classification, image processing, and machine learning.