Research on fusion technology based on low-light visible image and infrared image

Abstract. Image fusion technology usually combines information from multiple images of the same scene into a single image so that the fused image is often more informative than any source image. Considering the characteristics of low-light visible images, this study presents an image fusion technology to improve contrast of low-light images. This study proposes an adaptive threshold-based fusion rule. Threshold is related to the brightness distribution of original images. Then, the fusion of low-frequency coefficients is determined by threshold. Pulse-coupled neural networks (PCNN)-based fusion rule is proposed for fusion of high-frequency coefficients. Firing times of PCNN reflect the amount of detail information. Thus, a high-frequency coefficient corresponding to maximum firing times is chosen as the fused coefficient. Experimental results demonstrate that the proposed method obtains high-contrast images and outperforms traditional fusion approaches on image quality.


Introduction
Image fusion technology is aimed at obtaining one image with high quality from two more images of the same scene. 1Image fusion technology has been applied in various fields, such as remote sensing, 2 medical diagnosis, 3,4 face recognition, 5 target detection, 6,7 and art analysis. 8Fusion technology of infrared and visible images is an important branch in the image fusion field.An infrared image can display camouflaged objects under trees or smog.But its contrast is low and it is difficult to observe or recognize targets for the human vision system.On the contrary, a visible image usually contains much more texture and color information of the scene.But there are many dark regions in low-light visible images.Objects are fuzzy and difficult to be distinguished in these regions.However, fused images can effectively combine advantages of two images that can extend and enhance image information.Thus, it is necessary to study fusion technology of infrared and low-light visible images.
In the past few years, some techniques and software algorithms for image fusion have been developed. 9,10Generally, image fusion methods mainly include pixel-level fusion, feature-level fusion, and decision-level fusion.Compared to others, pixel-level fusion can better preserve original image information.In most pulse-coupled neural networks (PCNN)based fusion algorithms, only the single pixel value is used to motivate a PCNN neuron.It is not effective enough because human eyes are usually sensitive to features.Qu proposed an orientation information motivated PCNN algorithm (OI-PCNN).Orientation information is considered as a feature to motivate PCNN. 11This algorithm can preserve spatial characteristics of source images well and only grayscale source images are considered in this method.Kong et al. 12 proposed an adaptive fusion technique based on nonsubsampled contourlet transform and intensity-hue-saturation (IHS) transform.The pseudocolor principle is used in the fusion step.Fused images can obtain color information of reference images based on the pseudocolor principle.Results obtained show a good performance.But the algorithm costs too much time and pseudocolor information of targets may result in erroneous determination.Li et al. 13 proposed a guided filtering-based fusion method (GFF).First, source images are decomposed into a base layer containing large scale variations in intensity and a detail layer capturing small-scale details.Then a weighted average technique based on guided filtering is chosen for fusion of base and detail layers.Zhou and Tan 14 combined infrared and visible images using wavelet transform (WT).Li 15 proposed an infrared and visible image fusion algorithm based on target segmentation.First, a segmentation algorithm is used to extract target of an infrared image.Then, the target is fused with a visible image.This method preserves target information of infrared effectively.But the other information of the infrared image is lost.
A fused image retains most desirable information and characteristics of infrared and visible images.Generally, grayscale visible images in daytime are mostly used in traditional methods.Moreover, registration problems of infrared and visible images are not considered.Therefore, traditional fusion methods are not suitable for fusion of infrared and low-light visible images.To overcome these problems, a fusion technology is presented in this paper.It can be widely applied in various fields, such as target recognition, medical diagnosis, transportation, intelligent transportation, and video surveillance.
Necessary background information is provided in Sec. 2. In Sec. 3, the proposed image fusion method is described.Moreover, improved fusion rules are introduced.Experimental results are presented and discussed in Sec. 4. Finally, the conclusion of this study is presented in Sec. 5.

Background
Image fusion algorithms mainly include three categories, i.e., pixel-level fusion, feature-level fusion, and decision-level fusion.Compared with feature-level and decision-level fusion, pixel-level fusion directly combines the pixels of original images, which is applied more widely.
Multiscale transform methods have been demonstrated to be very useful in the pixel-level image fusion field.9][20] In addition to these, some multiscale transform methods are proposed, such as dualtree discrete WT (DT-DWT), 21 curvelet transform, 22 contourlet transform, 23 and so on.DWT or DT-DWT often suffers from limited directions and cannot represent edges of images.Moreover, the traditional DWT-based fusion method can introduce a blocking effect to fused image.
For representing images accurately, Candès and Donoho 24 proposed curvelet transform.Unlike curvelet transform that first develops a transform in the continuous domain and then discrete for sampled data, contourlet transform starts with a discrete-domain construction directly. 25Contourlet transform is composed of Laplacian pyramid and directional filter bank (DFB).Images can be decomposed into multiscales by Laplacian pyramid.Then DFBs are carried out to obtain directional components.The low-frequency part is always obtained by Laplacian pyramid decomposition.In addition, the high-frequency part is that the original image subtracts the low-frequency part, which is through upsampling.The Laplacian pyramid decomposition process is as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 6 3 ; E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 6 3 ; 3 7 6 G k ðm; nÞ ¼ E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 6 3 ; where k represents the decomposition levels, (m; n) is the pixel position of k'th level image, and wði; jÞ is the low-pass filter.When ðm − iÞ∕2 and ðn − jÞ∕2 are integer, The LP decomposition at each level generates a downsampled low-pass version of the original and the difference between the original and the prediction, resulting in a bandpass image. 25And each bandpass image is further decomposed by an l j level DFB into 2 lj bandpass directional images.For the traditional contourlet transform-based fusion method, low-frequency coefficients are fused by average rule and high-frequency coefficients are fused by absolute value choosing max rule.In general, it is difficult to obtain highcontrast and high-definition fused images.

Proposed Method
Original images have been strictly matched in most image fusion algorithms.But the registration problem is important in the image fusion field.In addition, a low-light visible image has many dark regions.These dark regions result in reducing quality of the fused image.Furthermore, it is difficult to observe objects in infrared images captured directly.A low-light visible image should also be transformed to an intensity image in the fusion step.
To overcome these problems, the proposed fusion method mainly includes preprocessed sections (such as IHS transform, original images registration, and image inverse), contourlet transform, coefficients fusion, and inverse contourlet transform.Figure 1 shows the structure of the proposed image fusion method.
According to the structure of the proposed image fusion method, the main steps are as follows: 1. Source images should be matched first.After that, matched images are enhanced to improve contrast.Then the luminance component of low-light visible image I v is extracted using IHS transform.2. After preprocessing, infrared and luminance images (I IR and I v are decomposed separately to lowfrequency coefficients (I J IR ; I J v ) and high-frequency coefficients (I d;k IR ; I d;k v ) using contourlet transform, respectively.J is the number of decomposition stages.k shows directional number at d'th scale (J ≥ d ≥ 1).Generally, more decomposition levels cost longer computation time.Therefore, two decomposition levels are chosen in this study.Generally, more directional subbands mean more high-frequency information.To reduce time consumption, two directional subbands are obtained in the first decomposition level and 16 directional subbands are obtained in the second level.

Low-Frequency Coefficients Fusion
Low-frequency coefficients mainly contain approximate characteristics of infrared and low-light visible images.First, threshold value is adjusted based on maximum of the illuminance image.Then, current pixel value of the illuminance image is used as the fused pixel if the difference is bigger than threshold through comparing.Otherwise, the average rule is used for low-frequency coefficients fusion.Threshold adjustment is one key step in low-frequency coefficients fusion.Mostly, there are only small amounts of brightest pixel values in low-light visible images.In lowlight visible images, few pixels have high intensity value.These high intensity pixels are mostly from background light, such as car light, lamp light, and other light-emitting devices.After two levels of decomposition, low-frequency coefficients (I J v ; I J IR ) are obtained.Through experiments, the top 0.13% largest values of coefficients difference are considered as background light components.Therefore, coefficients difference is sorted first.Then, the top 0.13% largest values are chosen to determine the threshold.In this paper, w th ¼ 0.75 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 6 3 ; 3 7 8 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 5 ; 6 3 ; 3 4 8 I low where I th is the threshold and w th is the threshold weight.I J IR and I J v are corresponding low-frequency coefficients.

High-Frequency Coefficients Fusion
High-frequency coefficients mainly contain detail characteristics of infrared and illuminance images.It represents texture and edge information of original images.The rule of maximum absolute value is usually chosen in most traditional fusion methods.But fused image quality is often bad.PCNN is a simplified neural network model, which is constructed by a plurality of interconnected neurons.Each pixel datum is one neuron in image processing and each neuron mainly includes dendritic branch, connector, and pulse generator sections.For high-frequency coefficients fusion, the mathematical model based on PCNN can be described as follows.High-frequency coefficients are used to motivate PCNN E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 6 3 ; 1 2 7 F d;k i;j ðn 0 Þ ¼ I d;k i;j ; where F d;k i;j is an output of feeding part, I d;k i;j is an input of feeding part, n 0 shows iteration time, k is the decomposition level, d represents direction at k 0 th level, and (i; j) shows pixel location.
Linking part is described as ; t e m p : i n t r a l i n k -; e 0 0 7 ; 3 2 6 ; 7 1 9 W ij;mn Y d;k ij;mn ðn 0 − 1Þ; (7)   where L d;k i;j ðn 0 Þ is an output of linking part, m and n are the scope of the connected neurons, V L is a normalization coefficient, and W ij;mn is the weight to connect other neurons E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 8 ; 3 2 6 ; 6 2 5 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 9 ; 3 2 6 ; 5 9 1 where U d;k i;j ðn 0 Þ is an internal state, θ d;k i;j ðn 0 Þ is a threshold, β, a θ , and V θ are the constant parameters.In each iteration, output (also called firing time) is calculated as 26 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 0 ; 3 2 6 ; 5 2 9 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 1 ; 3 2 6 ; 4 8 5 where X represents the low-light visible or infrared image and N is the total number of iteration times.
After N iterations, it is easy to obtain firing times of highfrequency coefficients.Then, fused high-frequency coefficients are obtained as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 2 ; 3 2 6 ; 3 8 5

Experimental Results and Discussion
To verify practical value of the proposed method, fused image quality and fusion effect are analyzed in this section.

Evaluation Criteria
Subjective criterion is mainly based on observation of human eyes.It is always hard to evaluate fused image quality.
Objective criteria are based on characteristics of fused images and are often applied to evaluate image quality.To verify validity of the proposed method, subjective criterion and objective criteria are both analyzed in this section.Mean value reflects average brightness of an entire image.Usually, it is hard to distinguish objects in a low luminance region for human eyes.Mean value represents average illumination of fused images.It is useful to evaluate the dynamic range of low-light fused images indirectly.For the human visual system, a high dynamic range of fused image brightness always indicates high definition.So larger mean value often shows better fused effect.
Standard deviation reflects gray-level distribution of a fused image.It describes the discrete degree between image pixels and mean value.Standard deviation is often used to evaluate contrast of fused images.The below equation shows the definition: where δ fused is the standard deviation of the fused image.Size of the fused image is m × n. pði; jÞ is the probability of the position (i; j).Mean value of the fused image is μ.
Entropy describes average information carried by a fused image.If entropy of a fused image is large, it indicates that there is a lot of information in the fused image.Definition is shown as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 4 ; 3 2 6 ; 5 6 8 where E fused is the entropy of the fused image and p j is the probability of pixel value j.L is 255, generally.Average gradient is also called image clarity.Generally, a larger value depicts better image quality.Equation (15)  shows the definition of average gradient 27 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 5 ; 6 3 ; 7 1 9 where g fused is the average gradient of the fused image and Iði; jÞ, is the pixel value at (i; j).Size of fused image is m × n.

Grayscale Images Fusion Analysis
To evaluate the proposed method, different fusion algorithms are chosen to compare.A fused image is mainly the fusion of intensity images and enhanced infrared images.Then, subjective criterion and objective criteria are used to analyze image quality.The first group experiment images are obtained from a website (Ref.28). Figure 2 shows original images with resolution 640 × 480.From the above red rectangle, clear words can be seen in Fig. 2(a) and nothing in Fig. 2(b).However, from the left red rectangle, a hidden person appears in Fig. 2(b) and nothing in Fig. 2

(a).
There is different information in the original low-light visible image and infrared image observed in Fig. 2. To improve image quality, various methods are used to combine original images.Figure 3 shows fused images using different methods.Figure 3(a) represents the fused result using WT with two levels.Figure 3(b) shows a fused image using WT with four levels.Figure 3(c) is the fused result using traditional contourlet transform.Figure 3(d) shows a fused image of the OI-PCNN algorithm. 11The fusion result of the GFF algorithm 13 is shown in Fig. 3(e).Figure 3(f) shows the fused result of the proposed method.
From Fig. 3, we can see that there is an obvious blocking effect in Figs.3(a) and 3(b).And there is low contrast in Fig. 3(c).It is difficult to observe objects by human eyes.Figure 3(d) shows a number of fused errors in many areas, such as billboard and car lights.Figure 3(f) retains more information of original images than Fig. 3(e).The proposed method outperforms the OI-PCNN algorithm and GFF algorithm in comparison.
To analyze fused image quality more accurately, this study compares objective criteria of fused images in Table 1.The fused image of the OI-PCNN algorithm has the largest mean value and entropy value in Table 1.But from Fig. 3(d), we can clearly see that there are lots of fused errors, which impact on people's subjective feelings seriously.Due to the effect of fused errors, it is difficult to observe objects for the human eye visual system.The fused image of the proposed method has a larger mean value and entropy than the others except for the OI-PCNN algorithm.Moreover, it has the largest standard deviation and average gradient.Its average running time is large but less than OI-PCNN.Considering all the characteristics, the fused image contains most detail information and highest contrast by using the proposed method than the others.From Fig. 4(a), we can clearly see that there are many dark areas.Opposite, many targets are clear in the infrared image as shown in Fig. 4(b).In the proposed image fusion method, source images are matched first.Then matched images are enhanced to improve contrast based on the human eye vision system.In addition, matched visible images should be transformed to obtain intensity images using IHS transform.
Figure 5 shows a fused color image using different algorithms.In Fig. 5(a), the brightness of the fused color image is low.It is hard to observe the scene information for human eyes.Blocking effect reduces fused image quality in Fig. 5(b).In Fig. 5(c), image contrast still needs to be enhanced.In Fig. 5(d), there are obvious fusion errors in the sky area.In addition, there is a black edge around the lamps in Fig. 5(e).Moreover, brightness still needs to be improved.Figure 5(f) has better image quality than others subjectively.For example, the contour of cars is clearer than others, especially car wheels.In addition to that, there is still more edge information in the distant building and trees retained in Fig. 5(f) than others.Table 2 shows objective criteria of various methods.By comparison, the proposed method has better mean value, standard deviation, entropy value, and average gradient value than others.Generally, larger objective criteria reflect better fusion results.In addition, its average running time is large but less than OI-PCNN.From Table 2, we can see the proposed method has better fusion results of color images than others.Moreover, objective evaluation and subjective evaluation reach the same conclusion.It is easy to conclude that the proposed method outperforms the OI-PCNN algorithm and GFF algorithm.
Figure 6 shows another group of source images.Both the low-light image and infrared image are 640 × 480.In Fig. 6(a), lots of targets are not clear (such as people or trees) in the low-light image.However, it is clear in the infrared image as we can see in Fig. 6(b).
Figure 7 shows fused results using different methods.Figures 7(a) and 7(b) represent low contrast.Figure 7(c) has low brightness.From Fig. 7(d), we can see that there are many fusion errors.In Fig. 7(e), there are black pixels around lamps.In addition, it also has low brightness.Compared to others, the proposed method obtains high contrast and brightness as shown in Fig. 7(f).
From Table 3, we can see that the proposed method has the largest mean value, standard deviation, and entropy.Moreover, it also has larger average gradient value than the GFF algorithm and its average running time is less than the OI-PCNN algorithm.Therefore the proposed method has better fusion results than others by comparing objective criteria.Experimental results show that the proposed method outperforms the OI-PCNN algorithm and GFF algorithm.

Conclusion
In this paper, a fusion method of infrared and low-light visible images is proposed.First, original infrared and visible images are preprocessed.In addition, different fusion rules are chosen based on characteristics of low-and high-frequency information.The low-frequency component is fused using the adaptive threshold-based rule and the high-frequency component is fused based on PCNN.Moreover, the proposed method is also applicable for fusion of color visible images and infrared images.Finally, experimental results show that this method improves fused image quality through subjective and objective evaluations.Though the proposed method effectively improves image quality, some objects (such as people) are not clear enough.In the future, target extraction and enhancement can be introduced into the image fusion algorithm to improve fused image quality further.

E
Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 3 ; 6 3 ; 5 9 0

Fig. 2
Fig. 2 Original images: (a) original low-light visible image and (b) original infrared image.

Fig. 4
Fig. 4 Original images: (a) original low-light visible image and (b) original infrared image.

4. 3
Color-Scale Images Fusion AnalysisIn this section, a color-scale low-light image and infrared image are used for analysis.The original low-light visible image is captured by a Nikon camera.Moreover, the original infrared image is captured by a Xenics Bobcat-640 camera.Two group images are obtained at the south of the first school building in Changchun University of Science and Technology.Size of source image is 640 × 480.

Fig. 6
Fig. 6 Original images: (a) original low-light visible image and (b) original infrared image.

Table 1
Objective criteria of grayscale fused images.Bold values are used to show the best quality of objective criterion.