Nonuniformity correction scheme considering the effects of internal thermal stray light

Abstract. A stray light compensated nonuniformity correction (SLCNUC) method is proposed for infrared (IR) cameras. The proposed approach formulates thermal stray light compensation functions according to the internal temperature changes of the IR camera. To derive the compensation functions, we analyze the variations of the NUC parameters over the entire range of internal temperatures and determine the function coefficients by using a least squares method. Compared with the existing NUC methods used in previous studies, the major advantage of the proposed method is its effective reduction of nonuniformity even in highly accumulated thermal stray light situations. Experiments and comparisons performed with real IR images show that the proposed method maintains lower spatial noise over a wide internal temperature range of the IR camera. It is shown that the proposed NUC achieves an 18 dB higher peak signal-to-noise ratio than that of the conventional reference-based NUC method.


Introduction
Typically, infrared (IR) cameras are composed of several optomechanical devices, electronic circuits, an IR detector, and optical lenses. Among these, the IR detector is one of the most important components in IR cameras. There are two principal methods for acquiring IR images by IR detectors: the scanned method and the focal plane array (FPA) method; in recent years, the FPA method has been more widely used. An FPA consists of a number of photodiode sets that convert thermal energy into electric signals inside the IR detector. 1,2 When an FPA is used in an IR detector, its output can have nonuniformity problems if the response characteristic of each photodiode is different. This causes fixed pattern noise (FPN) in IR images, degrading the image quality significantly. 3 One of the most widely used methods for removong the nonuniformity in an FPA is a two-point nonuniformity correction (NUC), which is a representative algorithm of reference-based NUC (RBNUC). It creates two uniform reference sources by using a blackbody and calibrates the gain and offset parameters to have the same output for the same input for all photodiodes arranged in the FPA. 4 However, assuming that the FPA output is completely corrected to have uniform output at some point, its residual nonuniformity (RNU) will become larger with the changes of the external environments of the FPA over time. 5 Electronic boards and lens-driving motors mounted inside the IR camera constantly emit heat while the IR camera is in use. For IR cameras mounted on aircraft, external heat is constantly introduced through conduction via aerodynamic heating while flying. Since an IR camera responds to heat energy, heat generated inside the IR camera is expressed as a type of thermal noise. This internal thermal noise has the same effects as thermal stray light in an IR optical system. Although the cold shield at the front of the FPA blocks thermal stray light introduced from outside the field-of-view, the cold shield or optical path can be heated, and this will generate spatial nonuniformity in the IR images. Changes of the temperature inside the IR cameras can also change the output characteristics of the readout IC (readout integrated circuit).
As a solution for the above problem, scene-based nonuniformity correction (SBNUC), [6][7][8][9][10][11] which has the advantage of NUC without interruption of IR camera operation by utilizing characteristics of the image, has been studied. SBNUC updates the NUC parameters iteratively through computation by utilizing the image's statistics, differences between currently inputted image frames, and the movements of the IR camera. However, existing SBNUC algorithms may not work in some applications such as IR camera-equipped aircraft at high altitudes of 45,000 feet or higher because it is very difficult to find any difference between image frames from the fixed IR camera with the small movements. In addition, since most targets displayed on the image frame are point targets, it is very difficult to perform an SBNUC with information derived from the shape and size of the targets. 12 Recently, Sozzi et al. 13 analyzed and modeled spatial nonuniformity thoroughly by assuming that the optomechanical system and the detector effect can be separated. Although this method can mitigate the spatial nonuniformity caused by optomechanical devices, the performance of the NUC depends on the correction parameters calculated by optical analysis software (OAS). Unfortunately, in real situations, the correction parameters calculated by OAS did not sufficiently take into account the variation of detector output over time.
Thus, this study proposes an NUC method referred to as stray light compensated nonuniformity correction (SLCNUC) that can maintain uniformity of IR images despite the internal thermal stray light from the long operation of IR cameras in the above-mentioned operational environment.
The proposed SLCNUC method derives compensation functions considering the thermal stray light effect between the internal temperature of the IR camera and the NUC parameters by using the least squares method.

Influence of Internal Thermal Stray Light on
Infrared Image Quality The conventional observation model assumes that the output of each photodiode in the FPA is linear; this can be expressed as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 3 2 6 ; 7 5 2 Y i;j ðnÞ ¼ a i;j X i;j ðnÞ þ b i;j : (1) Here, i and j refer to coordinate indices that designate the pixel's positions on the FPA, n is the frame number, X i;j ðnÞ is the IR radiation emitted from the scene, and a and b refer to the gain and offset components of the output characteristic, respectively. In addition, Y i;j ðnÞ is a digital image output value displayed to the user. However, in real environments, radiation energy is transferred to the FPA not only via IR radiation from the real scene but also from various sources such as thermal stray light generated by the internal components. Figure 1 shows an example of image degradation due to thermal stray light inside the IR camera. The red arrows in Fig. 1 indicate the areas where the IR image has been affected by internally generated heat. This phenomenon increases over time, increasing the RNU significantly.
To verify thermal stray light caused by heating elements inside the IR camera, simulations were conducted using ASAP ™ , an optical ray-tracing-analysis software, as shown in Fig. 2. Figure 2(a) shows the main parts of the IR camera that can generate internal thermal noise, such as motors and cooler. Figure 2(b) shows the ray tracing result of each thermal noise source. Here, the scattering characteristics of the optical equipment were modeled using the Lambertian model and the Harvey bidirectional scatter distribution function model; 14 the results are shown in Table 1. The simulation showed that the maximum radiation intensity caused by the detector cooler, but arriving at the FPA, was 3.9 × 10 −2 times the emission intensity at the detector cooler, proving that the thermal noise emitted from the heating elements inside the IR camera affects the FPA.
To compare with the previous simulation result, an experiment was conducted using image frames from real IR cameras. We investigated the variation of spatial noise according to the changes of the internal temperature. The spatial noise was calculated using the standard deviation of the temporal averaged image between 300 consecutive nonuniformity-corrected image frames at every single internal temperature of the IR cameras. The results showed that the spatial noise increased linearly as a function of the internal  Kim and Chang: Nonuniformity correction scheme considering the effects of internal thermal stray light temperature, and there were no statistically significant differences according to the detectors or optical flows. Figure 3(a) shows an example of the linear relationship between spatial noise and internal temperature variation due to the long IR camera operating time. It is observed that the spatial noise increases almost in proportion to the increase of the internal temperature, where the second-order or any higher-order of nonlinear terms can be considered as negligibly small in this increase. The increase of the spatial noise is caused by the deviation of gain and offset. In other words, as shown in Fig. 3(b), the slope of the detector output curve of the pixels affected by the internal thermal stray light is increased in proportion to the increase in the IR camera internal temperature from T 1 to T 2 and T 3 . At the same time, since the slope of the NUC fitting line at each internal temperature changes according to the detector output curve, the gain and offset are also changed in proportion thereto. Therefore, the approximation of gain and offset with linear function of internal temperature can be regarded as the most general form for excluding high-order nonlinear term, since the spatial noise increases almost linearly with the internal temperature. We can conclude that the internal thermal stray light changes the internal temperature of the IR camera, and this variation of internal temperature, in turn, causes variation in the spatial noise of the IR camera. In addition, the variation of gain and offset due to the variation of spatial noise can be approximated by linear functions of the internal temperature.
Considering both the stray-light-tracing simulation result and the results of the actual experiments in a real IR environment, the conventional observation model should be modified as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 3 2 6 ; 6 6 4 Y i;j ðnÞ ¼ a i;j ðTÞ½X i;j ðnÞ þX i;j ðnÞ þ b i;j ðTÞ: (2) Here,X i;j ðnÞ refers to radiation energy from the actual target whileX i;j ðnÞ refers to the component seen by the FPA with the internal thermal stray light. The gain and offset, a and b, respectively, of the detector output are expressed as a linear function of the IR camera internal temperature T.
Equation (2) can be rearranged and summarized as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 3 2 6 ; 5 4 9X In the above equation, g and o are values of gain and offset used for output correction and these values change linearly as a function of the internal temperature of the IR camera.
That is, the observation model, which is a combined type of thermal stray light component due to IR camera internal temperature added to true scene value radiated from the target, can be produced by an estimation of gain and offset values that change according to the internal temperatures.

Proposed Calibration Method
In this section, the proposed SLCNUC method is formulated by providing a calibration process based on the estimation of the true scene irradiance. The proposed method of true scene Step motor 1 1.7 1.25 × 10 −5 Step motor 2 1.2 1.26 × 10 −6 DC motor 2 0 Fig. 3 Variation of spatial noise with the increase of the internal temperature of the IR camera: (a) experimental example of linear relationship between the spatial noise and the internal temperature and (b) slope changes of detector output curve and NUC fitting line with increasing internal temperature.
irradiation estimation is described in Sec. 3.1. The formulated SLCNUC method is described in Sec. 3.2.

Estimation of True Scene Irradiance
As demonstrated in Eq. (3), the nonuniformity problem deepens over time because of the presence of the internal thermal stray light termX i;j ðnÞ, which changes according to the internal temperature of the IR camera. This is because the existing RBNUC algorithm creates gain and offset by estimation based on the assumption that only true scene irradiance is present.
To estimate the true scene irradiance in the existing RBNUC methods, statistical methods such as the median or mean were applied to observe the IR image data. However, since locally heated areas without uniform heating can be present in thermal stray light environments inside the IR camera, statistical values such as the median value cannot estimate the true scene value. Therefore, this study calculated the true scene irradiance of the internal thermal stray light environments based on the amount of electric charge collected in the capacitor under an IR detector standalone condition. Assuming that the expectation value of true scene irradiance is Z in Eq. (4): In other words, the calculation of gain and offset in the SLCNUC method under the internal thermal stray light environment can be accomplished by estimating the true scene irradiance Z according to the changes in target temperature and measuring the changes in observation value Y according to the changes in IR camera internal temperatures. Figure 4 shows a graph of the number of electrons collected in a capacitor as a function of the blackbody temperature detected by the IR detector. The experimental environment is as follows: the FPA well fill capacity is 6.5 Me − and the integration time is 1 ms. The charge amount increased monotonically with the blackbody temperature. Equation (5) shows the estimation process of true scene irradiance based on the charge amount of the capacitor E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 5 ; 3 2 6 ; 4 2 2 F refers to the floor level value of the IR detector, i.e., the pixel value displayed even when the integration time is at its minimum value. C refers to the amount of electron charge acquired during an integration time of 1 ms, T int is the integration time, and DR is the dynamic range of the IR detector. In addition, K refers to the true scene gain, i.e., the number of electrons required to raise a pixel by one gray level. The result of true scene irradiance estimation under the detector standalone condition using Eq. (5) is shown in Table 2.
Here, 2 ms was used as the integration time T int , and the parameter values of 3726 and 21500 were assumed for F Fig. 4 The number of electrons collected in a capacitor as a function of blackbody temperature. and DR, respectively; these values had been supplied by the manufacturer.

Calibration Process
To calculate the NUC parameters in SLCNUC, gain and offset in Eq. (3), a range of IR camera internal temperatures was set. The range was divided into a number of sections for sampling, and the internal temperature T matrix was selected as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 6 3 ; 6 5 9 T ¼ ½T 1 ; T 2 ; T 3 ; : : : ; T 7 : For the gain and offset for all pixels at each internal temperature used in the T matrix, the true scene irradiance values calculated in Sec. 3.1 were used for the calculation. For example, when the internal temperature was T 1 , the calculation of gain and offset at a pixel with coordinates (i; j) was performed using Here, Z H refers to the estimated true scene irradiance acquired through Eq. (5) irradiated from the high-temperature blackbody, and Z L refers to estimated true scene irradiance acquired through Eq. (5) irradiated from the low-temperature blackbody. Y T 1 ;H;ði;jÞ and Y T 1 ;L;ði;jÞ refer to an observation model at the (i; j) coordinate acquired while looking at the high-and low-temperature blackbodies, respectively, when the IR camera's internal temperature was T 1 . Finally, gain and offset at each pixel under internal temperatures of other elements in the T matrix are acquired using Eq. (7), so the gain matrix and the offset matrix of each pixel according to a change in each internal temperature can be obtained as follows: A change in the NUC parameters according to the IR camera's internal temperature can be observed through the changes in g ði;jÞ and o ði;jÞ in each pixel, and this change can be expressed via a linear function of internal temperature such as G ¼ c 0 þ c 1 T by using the least squares method. 15 Here, T is the IR camera's internal temperature. The linear function can be expressed as ,TC ¼Ḡ and the coefficient matrixC of the linear function for changes in the gain and offset can be calculated by Eq. (9). The change function of the offset can be obtained in the same manner as that of the gain above E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 9 ; 6 3 ; 1 2 7C In the existing NUC, each pixel had a pair of values corresponding to gain and offset. However, in the proposed SLCNUC, a total of four coefficients are present in the linear stray light compensation function that defines the changes caused by the IR camera's internal temperature (two coefficients for each gain and offset). By inserting the internal temperature into the gain function and the offset function, gain and offset are calculated for use. As the camera's internal temperature does not change rapidly, there will be no significant computation increase or processing speed difference compared to the existing RBNUC method by employing periodic calculation of entire pixels and table updates accordingly. Furthermore, the proposed SLCNUC method does not require any kind of OAS or climatic chamber to calculate the NUC parameters.

Experiments for Performance Evaluation
To verify the performance of the proposed SLCNUC method, experiments performed with real IR images were conducted. The first experiment was a comparison of the spatial noise between the existing RBNUC and the proposed SLCNUC methods according to the IR camera's internal temperature. The internal temperature was measured by using temperature sensors attached to the optomechanics housing.
When the IR camera's internal temperature was set at 31°C 36°C, and 45°C, and the temperature of the blackbody was set to 20°C for low temperature and 60°C for high temperature, a gain/offset table using 2-point NUC was obtained. Then, changes in spatial noise according to the IR camera's internal temperature were observed. A Sebastian640 from the SCD Company was used as an IR detector, the wavelength was 3.6 to 4.9 μm for midwavelength infrared, and the resolution of the FPA was 640 × 512. Furthermore, a heater was installed inside the IR camera to raise the internal temperature quickly.
The experimental result of Fig. 5 shows that an IR camera, to which an NUC table acquired when the IR camera internal temperature was 31°C was applied, had the lowest spatial noise at an internal temperature of 31°C, which was the same as the temperature at acquisition. However, as the internal temperature increased, spatial noise also increased rapidly. On the other hand, an IR camera with NUC parameters acquired when a camera internal temperature of 45°C was applied showed the lowest spatial noise at high temperatures. However, it showed the highest spatial noise when the internal camera temperature was low. Meanwhile, an IR camera to which the proposed SLCNUC method was applied showed that not only a low level of spatial noise can be maintained at both high and low camera internal temperatures but also that spatial noise caused by changes in internal temperatures increased linearly. This linear increase in spatial noise can be explained by an increase in the number of defective pixels or an increase in the dark current of the detector due to long operation hours of the IR camera, which can be reduced through additional IR image correction techniques, such as defective pixel replacement.
The second experiment was conducted to verify how pixel output can be changed according to the changes in the IR camera's internal temperature while four arbitrary pixel coordinates were designated in the IR image frame. When the IR camera's internal temperature was 31°C, gain and offset were acquired and used in NUC. The internal temperature was increased constantly using a heater mounted inside the IR camera. As shown in Fig. 6(a), the IR camera, with the existing RBNUC algorithm applied, had different slopes of pixel value increase according to the pixel location with the increase of the internal camera temperature. This was because the effects of thermal stray light inside the IR camera can be different according to FPA locations and the nonuniformity can deepen as the camera operation time becomes longer, with thermal stray light that has different effects at different locations. On the other hand, an IR camera with the proposed SLCNUC method applied had uniform pixel value rise slopes at all FPA locations despite the temperature changes inside the IR camera, as shown in Fig. 6(b). This result implies that even though the spatial noise increases as the internal temperature rises, the pixel values of all the coordinates are increased to the same level, so the uniformity of IR image can be maintained.
The third experiment compared the performance by measuring peak signal-to-noise ratio (PSNR). To calculate the PSNR, the original image was defined as the one that is captured from the IR camera with an internal temperature of 46°C and corrected by an NUC table acquired at an internal temperature 46°C. Similarly, the other images were obtained by applying NUC tables acquired at different internal temperatures.
The experimental result showed that lower PSNR was measured when the internal temperature was cooled down   7 Comparison of PSNR performance according to NUC parameter acquired internal temperature: (a) IR image using NUC parameter acquired when internal temperature was 40°C, (b) IR image using NUC parameter acquired when internal temperature was 35°C, (c) IR image using NUC parameter acquired when internal temperature was 30°C, and (d) IR image using the SLCNUC method. from 46°C, as shown in Figs. 7(a)-7(c). Despite the internal temperature of the IR camera at 46°C, when correction parameters acquired at different internal temperatures were used, nonuniformity deepened. On the contrary, when an IR image was shot by applying the proposed SLCNUC method, PSNR was measured at 65.1 dB, which was the highest performance, as shown in Fig. 7(d). When the region marked with the red-colored square in Fig. 7 was compared, the detail of the IR image to which the proposed SLCNUC method was applied was the closest to the original image.

Conclusions
An NUC scheme named SLCNUC was proposed. Our work formulated the linear stray light compensation function to correct the local nonuniformity caused by internal thermal noise. The conventional NUC algorithms have difficulties in correcting the thermal stray light inside the IR camera because the thermal stray light is highly accumulated by means of the internal thermal noise through long operating hours and has complex effects that depend on the design of the optical system. In this study, the thermal stray light inside the IR camera was analyzed to prove that the internal thermal noise can affect the IR image quality, and a linear function was derived to compensate internal thermal noise iteratively. Experimental results on real IR images demonstrated that the proposed SLCNUC could maintain the lower level of spatial noise in spite of higher levels of thermal stray light and had better image quality with up to 18 dB PSNR improvement compared to that of the existing RBNUC method.