Typically, infrared (IR) cameras are composed of several optomechanical devices, electronic circuits, an IR detector, and optical lenses. Among these, the IR detector is one of the most important components in IR cameras. There are two principal methods for acquiring IR images by IR detectors: the scanned method and the focal plane array (FPA) method; in recent years, the FPA method has been more widely used. An FPA consists of a number of photodiode sets that convert thermal energy into electric signals inside the IR detector.1,2 When an FPA is used in an IR detector, its output can have nonuniformity problems if the response characteristic of each photodiode is different. This causes fixed pattern noise (FPN) in IR images, degrading the image quality significantly.3
One of the most widely used methods for removong the nonuniformity in an FPA is a two-point nonuniformity correction (NUC), which is a representative algorithm of reference-based NUC (RBNUC). It creates two uniform reference sources by using a blackbody and calibrates the gain and offset parameters to have the same output for the same input for all photodiodes arranged in the FPA.4 However, assuming that the FPA output is completely corrected to have uniform output at some point, its residual nonuniformity (RNU) will become larger with the changes of the external environments of the FPA over time.5
Electronic boards and lens-driving motors mounted inside the IR camera constantly emit heat while the IR camera is in use. For IR cameras mounted on aircraft, external heat is constantly introduced through conduction via aerodynamic heating while flying. Since an IR camera responds to heat energy, heat generated inside the IR camera is expressed as a type of thermal noise. This internal thermal noise has the same effects as thermal stray light in an IR optical system. Although the cold shield at the front of the FPA blocks thermal stray light introduced from outside the field-of-view, the cold shield or optical path can be heated, and this will generate spatial nonuniformity in the IR images. Changes of the temperature inside the IR cameras can also change the output characteristics of the readout IC (readout integrated circuit).
As a solution for the above problem, scene-based nonuniformity correction (SBNUC),220.127.116.11.–11 which has the advantage of NUC without interruption of IR camera operation by utilizing characteristics of the image, has been studied. SBNUC updates the NUC parameters iteratively through computation by utilizing the image’s statistics, differences between currently inputted image frames, and the movements of the IR camera. However, existing SBNUC algorithms may not work in some applications such as IR camera-equipped aircraft at high altitudes of 45,000 feet or higher because it is very difficult to find any difference between image frames from the fixed IR camera with the small movements. In addition, since most targets displayed on the image frame are point targets, it is very difficult to perform an SBNUC with information derived from the shape and size of the targets.12 Recently, Sozzi et al.13 analyzed and modeled spatial nonuniformity thoroughly by assuming that the optomechanical system and the detector effect can be separated. Although this method can mitigate the spatial nonuniformity caused by optomechanical devices, the performance of the NUC depends on the correction parameters calculated by optical analysis software (OAS). Unfortunately, in real situations, the correction parameters calculated by OAS did not sufficiently take into account the variation of detector output over time.
Thus, this study proposes an NUC method referred to as stray light compensated nonuniformity correction (SLCNUC) that can maintain uniformity of IR images despite the internal thermal stray light from the long operation of IR cameras in the above-mentioned operational environment. The proposed SLCNUC method derives compensation functions considering the thermal stray light effect between the internal temperature of the IR camera and the NUC parameters by using the least squares method.
Influence of Internal Thermal Stray Light on Infrared Image Quality
The conventional observation model assumes that the output of each photodiode in the FPA is linear; this can be expressed as follows:Figure 1 shows an example of image degradation due to thermal stray light inside the IR camera. The red arrows in Fig. 1 indicate the areas where the IR image has been affected by internally generated heat. This phenomenon increases over time, increasing the RNU significantly.
To verify thermal stray light caused by heating elements inside the IR camera, simulations were conducted using ASAP™, an optical ray-tracing-analysis software, as shown in Fig. 2. Figure 2(a) shows the main parts of the IR camera that can generate internal thermal noise, such as motors and cooler. Figure 2(b) shows the ray tracing result of each thermal noise source. Here, the scattering characteristics of the optical equipment were modeled using the Lambertian model and the Harvey bidirectional scatter distribution function model;14 the results are shown in Table 1. The simulation showed that the maximum radiation intensity caused by the detector cooler, but arriving at the FPA, was times the emission intensity at the detector cooler, proving that the thermal noise emitted from the heating elements inside the IR camera affects the FPA.
Ray-tracing simulation result of internal thermal noise.
|Internal thermal noise source||Power consumption (W)||Radiation intensity reaching the FPA compared to level at its source|
|Step motor 1||1.7||1.25×10−5|
|Step motor 2||1.2||1.26×10−6|
To compare with the previous simulation result, an experiment was conducted using image frames from real IR cameras. We investigated the variation of spatial noise according to the changes of the internal temperature. The spatial noise was calculated using the standard deviation of the temporal averaged image between 300 consecutive nonuniformity-corrected image frames at every single internal temperature of the IR cameras. The results showed that the spatial noise increased linearly as a function of the internal temperature, and there were no statistically significant differences according to the detectors or optical flows. Figure 3(a) shows an example of the linear relationship between spatial noise and internal temperature variation due to the long IR camera operating time. It is observed that the spatial noise increases almost in proportion to the increase of the internal temperature, where the second-order or any higher-order of nonlinear terms can be considered as negligibly small in this increase. The increase of the spatial noise is caused by the deviation of gain and offset. In other words, as shown in Fig. 3(b), the slope of the detector output curve of the pixels affected by the internal thermal stray light is increased in proportion to the increase in the IR camera internal temperature from to and . At the same time, since the slope of the NUC fitting line at each internal temperature changes according to the detector output curve, the gain and offset are also changed in proportion thereto. Therefore, the approximation of gain and offset with linear function of internal temperature can be regarded as the most general form for excluding high-order nonlinear term, since the spatial noise increases almost linearly with the internal temperature.
We can conclude that the internal thermal stray light changes the internal temperature of the IR camera, and this variation of internal temperature, in turn, causes variation in the spatial noise of the IR camera. In addition, the variation of gain and offset due to the variation of spatial noise can be approximated by linear functions of the internal temperature.
Considering both the stray-light-tracing simulation result and the results of the actual experiments in a real IR environment, the conventional observation model should be modified as follows:
Equation (2) can be rearranged and summarized as follows:
Proposed Calibration Method
In this section, the proposed SLCNUC method is formulated by providing a calibration process based on the estimation of the true scene irradiance. The proposed method of true scene irradiation estimation is described in Sec. 3.1. The formulated SLCNUC method is described in Sec. 3.2.
Estimation of True Scene Irradiance
As demonstrated in Eq. (3), the nonuniformity problem deepens over time because of the presence of the internal thermal stray light term , which changes according to the internal temperature of the IR camera. This is because the existing RBNUC algorithm creates gain and offset by estimation based on the assumption that only true scene irradiance is present.
To estimate the true scene irradiance in the existing RBNUC methods, statistical methods such as the median or mean were applied to observe the IR image data. However, since locally heated areas without uniform heating can be present in thermal stray light environments inside the IR camera, statistical values such as the median value cannot estimate the true scene value. Therefore, this study calculated the true scene irradiance of the internal thermal stray light environments based on the amount of electric charge collected in the capacitor under an IR detector standalone condition. Assuming that the expectation value of true scene irradiance is in Eq. (4):
Figure 4 shows a graph of the number of electrons collected in a capacitor as a function of the blackbody temperature detected by the IR detector. The experimental environment is as follows: the FPA well fill capacity is and the integration time is 1 ms. The charge amount increased monotonically with the blackbody temperature. Equation (5) shows the estimation process of true scene irradiance based on the charge amount of the capacitorTable 2. Here, 2 ms was used as the integration time , and the parameter values of 3726 and 21500 were assumed for and DR, respectively; these values had been supplied by the manufacturer.
True scene irradiance estimation according to blackbody temperature.
|Blackbody temperature (°C)||Charge (Me)||Estimated pixel value|
To calculate the NUC parameters in SLCNUC, gain and offset in Eq. (3), a range of IR camera internal temperatures was set. The range was divided into a number of sections for sampling, and the internal temperature matrix was selected as3.1 were used for the calculation. For example, when the internal temperature was , the calculation of gain and offset at a pixel with coordinates () was performed using
Finally, gain and offset at each pixel under internal temperatures of other elements in the matrix are acquired using Eq. (7), so the gain matrix and the offset matrix of each pixel according to a change in each internal temperature can be obtained as follows:15 Here, is the IR camera’s internal temperature. The linear function can be expressed as , and the coefficient matrix of the linear function for changes in the gain and offset can be calculated by Eq. (9). The change function of the offset can be obtained in the same manner as that of the gain above
In the existing NUC, each pixel had a pair of values corresponding to gain and offset. However, in the proposed SLCNUC, a total of four coefficients are present in the linear stray light compensation function that defines the changes caused by the IR camera’s internal temperature (two coefficients for each gain and offset). By inserting the internal temperature into the gain function and the offset function, gain and offset are calculated for use. As the camera’s internal temperature does not change rapidly, there will be no significant computation increase or processing speed difference compared to the existing RBNUC method by employing periodic calculation of entire pixels and table updates accordingly. Furthermore, the proposed SLCNUC method does not require any kind of OAS or climatic chamber to calculate the NUC parameters.
Experiments for Performance Evaluation
To verify the performance of the proposed SLCNUC method, experiments performed with real IR images were conducted. The first experiment was a comparison of the spatial noise between the existing RBNUC and the proposed SLCNUC methods according to the IR camera’s internal temperature. The internal temperature was measured by using temperature sensors attached to the optomechanics housing.
When the IR camera’s internal temperature was set at 31°C36°C, and 45°C, and the temperature of the blackbody was set to 20°C for low temperature and 60°C for high temperature, a gain/offset table using 2-point NUC was obtained. Then, changes in spatial noise according to the IR camera’s internal temperature were observed. A Sebastian640 from the SCD Company was used as an IR detector, the wavelength was 3.6 to for midwavelength infrared, and the resolution of the FPA was . Furthermore, a heater was installed inside the IR camera to raise the internal temperature quickly.
The experimental result of Fig. 5 shows that an IR camera, to which an NUC table acquired when the IR camera internal temperature was 31°C was applied, had the lowest spatial noise at an internal temperature of 31°C, which was the same as the temperature at acquisition. However, as the internal temperature increased, spatial noise also increased rapidly. On the other hand, an IR camera with NUC parameters acquired when a camera internal temperature of 45°C was applied showed the lowest spatial noise at high temperatures. However, it showed the highest spatial noise when the internal camera temperature was low. Meanwhile, an IR camera to which the proposed SLCNUC method was applied showed that not only a low level of spatial noise can be maintained at both high and low camera internal temperatures but also that spatial noise caused by changes in internal temperatures increased linearly. This linear increase in spatial noise can be explained by an increase in the number of defective pixels or an increase in the dark current of the detector due to long operation hours of the IR camera, which can be reduced through additional IR image correction techniques, such as defective pixel replacement.
The second experiment was conducted to verify how pixel output can be changed according to the changes in the IR camera’s internal temperature while four arbitrary pixel coordinates were designated in the IR image frame. When the IR camera’s internal temperature was 31°C, gain and offset were acquired and used in NUC. The internal temperature was increased constantly using a heater mounted inside the IR camera. As shown in Fig. 6(a), the IR camera, with the existing RBNUC algorithm applied, had different slopes of pixel value increase according to the pixel location with the increase of the internal camera temperature. This was because the effects of thermal stray light inside the IR camera can be different according to FPA locations and the nonuniformity can deepen as the camera operation time becomes longer, with thermal stray light that has different effects at different locations. On the other hand, an IR camera with the proposed SLCNUC method applied had uniform pixel value rise slopes at all FPA locations despite the temperature changes inside the IR camera, as shown in Fig. 6(b). This result implies that even though the spatial noise increases as the internal temperature rises, the pixel values of all the coordinates are increased to the same level, so the uniformity of IR image can be maintained.
The third experiment compared the performance by measuring peak signal-to-noise ratio (PSNR). To calculate the PSNR, the original image was defined as the one that is captured from the IR camera with an internal temperature of 46°C and corrected by an NUC table acquired at an internal temperature 46°C. Similarly, the other images were obtained by applying NUC tables acquired at different internal temperatures.
The experimental result showed that lower PSNR was measured when the internal temperature was cooled down from 46°C, as shown in Figs. 7(a)–7(c). Despite the internal temperature of the IR camera at 46°C, when correction parameters acquired at different internal temperatures were used, nonuniformity deepened. On the contrary, when an IR image was shot by applying the proposed SLCNUC method, PSNR was measured at 65.1 dB, which was the highest performance, as shown in Fig. 7(d). When the region marked with the red-colored square in Fig. 7 was compared, the detail of the IR image to which the proposed SLCNUC method was applied was the closest to the original image.
An NUC scheme named SLCNUC was proposed. Our work formulated the linear stray light compensation function to correct the local nonuniformity caused by internal thermal noise. The conventional NUC algorithms have difficulties in correcting the thermal stray light inside the IR camera because the thermal stray light is highly accumulated by means of the internal thermal noise through long operating hours and has complex effects that depend on the design of the optical system. In this study, the thermal stray light inside the IR camera was analyzed to prove that the internal thermal noise can affect the IR image quality, and a linear function was derived to compensate internal thermal noise iteratively. Experimental results on real IR images demonstrated that the proposed SLCNUC could maintain the lower level of spatial noise in spite of higher levels of thermal stray light and had better image quality with up to 18 dB PSNR improvement compared to that of the existing RBNUC method.
W. Zhao and C. Zhang, “Scene-based nonuniformity correction and enhancement: pixel statistics and subpixel motion,” J. Opt. Soc. Am. A 25(7), 1668–1681 (2008).http://dx.doi.org/10.1364/JOSAA.25.001668Google Scholar
S. N. Torres et al., “Adaptive scene-based nonuniformity correction method for infrared-focal plane arrays,” Proc. SPIE 5076, 130 (2003).PSISDG0277-786Xhttp://dx.doi.org/10.1117/12.487217Google Scholar
R. C. Hardie et al., “Scene-based nonuniformity correction with reduced ghosting using a gated LMS algorithm,” Opt. Express 17(17), 14918–14933 (2009).OPEXFF1094-4087http://dx.doi.org/10.1364/OE.17.014918Google Scholar
D. A. Scribner et al., “Adaptive nonuniformity correction for IR focal-plane arrays using neural networks,” Proc. SPIE 1541, 100–109 (1991).PSISDG0277-786Xhttp://dx.doi.org/10.1117/12.49324Google Scholar
J. G. Harris and Y.-M. Chiang, “Nonuniformity correction of infrared image sequences using the constant-statistics constraint,” IEEE Trans. Image Process. 8(8), 1148–1151 (1999).IIPRE41057-7149http://dx.doi.org/10.1109/83.777098Google Scholar
R. C. Hardie et al., “Scene-based nonuniformity correction with video sequences and registration,” Appl. Opt. 39(8), 1241–1250 (2000).APOPAI0003-6935http://dx.doi.org/10.1364/AO.39.001241Google Scholar
J. Hu, Z. Xu and Q. Wan, “Non-uniformity correction of infrared focal plane array in point target surveillance systems,” Infrared Phys. Technol. 66, 56–69 (2014).IPTEEY1350-4495http://dx.doi.org/10.1016/j.infrared.2014.05.012Google Scholar
B. Sozzi et al., “Thermal imager sources of non-uniformities: modeling of static and dynamic contributions during operations,” Proc. SPIE 9071, 907104 (2014).PSISDG0277-786Xhttp://dx.doi.org/10.1117/12.2049914Google Scholar
E. Fest, “Stray light control for molded optics,” in Molded Optics: Design and Manufacture, and M. Schaub et al., Eds., pp. 72–125, CRC Press, Boca Raton, Florida (2011).Google Scholar
Shinwook Kim received his BS and MS degrees in electrical and electronics engineering from Chung-Ang University in 2008 and 2010, respectively. He is currently a PhD candidate in the Department of Electrical and Electronics Engineering at Chung-Ang University in Seoul, Republic of Korea. His research interests are in the areas of digital signal processing and infrared imaging systems.
Tae-Gyu Chang received his BS degree from Seoul National University in 1979, his MS degree from Korea Advanced Institute of Science and Technology in 1981, and his PhD from the University of Florida, Gainesville, in 1987, all in electrical engineering. In 1990, he joined the faculty of Chung-Ang University, Seoul, where he is currently a professor in the Department of Electrical and Electronics Engineering. His research interests include adaptive signal processing and communications.