Camera calibration method in specific bands for the near-infrared dynamic navigator

Abstract. The positional accuracy of a near-infrared (NIR) dynamic navigator is remarkably affected by two factors. One is the calibration accuracy of the navigator’s two NIR cameras, and the other is the accuracy of feature point extraction. The current lack of accurate calibration devices for NIR cameras limits further application. Therefore, in this study, an NIR camera calibration device was designed by placing the NIR light source and heat dissipation system at the back of a Halcon transparent glass template. Usage of camera calibration in a specific band and the gray centroid method based on elliptic boundary to extract feature points can further improve the accuracy of vision system calibration and measurement. Repeated tests and verifications showed that the reconstruction accuracy (<0.1  pixels) of the binocular vision system calibrated by the NIR calibration device in a specific band was better than that calibrated by traditional methods.


Introduction
Near-infrared (NIR) dynamic navigators are widely used in surgery due to their accurate real-time tracking and positioning. [1][2][3] Their basic principle is binocular stereo vision that uses a visualization system of two NIR cameras. Camera imaging in NIR band can remarkably reduce the impact of environmental noise on measurement results and decrease the complexity of feature extraction. However, due to the lack of high-precision NIR calibration devices, cameras are usually calibrated under the visible light band, which in turn, adversely affects the accuracy of real-time tracking and positioning. Thus, accurate calibration of NIR cameras is important in improving the accuracy of NIR navigation systems.
In contrast to visible cameras, IR cameras cannot clearly capture feature points of the commonly used visible light calibration templates. Thus, high-precision calibration under IR remains a challenge and restricts the application of IR cameras. 4 The calibration of IR cameras is a highly active research field and has achieved certain developments in past decades.
IR calibration methods can be classified into three main types. First is the heating method, in which the calibration template is heated to obtain an IR image. [5][6][7][8] For example, Hilsenstein 7 printed a checkerboard pattern onto a circuit board. A high-contrast IR image was generated after being heated by a hairdryer because the emissivity of the substrate material and copper was inconsistent. This type of method is widely used, but its main disadvantage is low precision due to low mechanical precision of the self-made calibration template, thermal expansion, and material contraction.
The second type is the active illuminating method. Bai-Sheng et al. 9 used NIR LED on a calibration template to obtain an NIR calibration template. The binocular vision of visible light was used to obtain the geometric position of LED, and the geometrical information of the calibration template was obtained. Calibration templates of the same idea have been widely used in the calibration of NIR cameras. [10][11][12][13] However, serious shortcomings are observed. First, the center of the target lamps cannot be accurately measured. Second, the lamps are generally placed in the holes of the support plate, thus obtaining high mounting precision is difficult.
The third type is the mask method, in which an IR calibration image is obtained by placing a mask before the active heating element. The checkerboard calibration template heating specific areas are used to obtain a clear IR image in the method. [14][15][16] However, this method reduces the accuracy of the IR calibration template due to uneven heating and thermal expansion.
In this study, an NIR camera calibration device in a specific band was designed to overcome the shortcomings of existing methods. To improve the accuracy of feature point extraction, the algorithm was improved by combining elliptic fitting with gray centroid method. Considering the influence of chromatic aberration on lenses, the camera calibration in a specific band is introduced to further improve the calibration accuracy.
The rest of the paper is divided into the following sections. Section 2 introduces basic theories of binocular vision calibration and chromatic aberration effect of lens. Section 3 presents the design of the NIR calibration device. Section 4 discusses the extraction of feature points in the calibration device and verifies the accuracy through experiments. The final section provides conclusions.

Binocular Vision Calibration
Binocular vision calibration not only needs to obtain the intrinsic and extrinsic parameters and the distortion coefficients of each camera but also needs to determine the relative position between two cameras. 17 In binocular vision calibration, two cameras can photograph a calibration device at the same time as one corresponding image pair. The world coordinate systems are then unified through the corresponding relation of feature points.
The conversion relationship between the world and the camera coordinate systems in the camera model is generally expressed as follows.
Left camera: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 6 3 ; 6 3 1 Right camera: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 6 3 ; 5 6 5 where R r , T r , R l , T l are the extrinsic parameters (rotation and translation) of the right and left cameras, respectively. Equations (1) and (2) are then converted into the following representation: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 6 3 ; 4 6 6 " x r y r z r where rotation (R) and translation (T) parameters of the right camera relative to the left camera can be expressed as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 3 2 6 ; 7 5 2 The calculation of R and T is used to unify the left and right cameras to the same world coordinate system, which is shown in Fig. 1.

Chromatic Aberration Effect of Lens
Different focal lengths are available for the same lens due to the chromatic aberration effect. The basic relationship indicates that the focal length is long when the light wavelength is long. A camera lens is composed of a series of lens groups. Although correction of chromatic aberration is strictly considered in the manufacturing process, internal parameters of the camera are expected to be affected by the working band. For example, visible-light and NIR bands are included in the sensitization range for commonly used industrial cameras in the market. If the calibration parameters of the visible-light band are used for the camera working in the NIR band, then large systematic errors occur. This scenario is tested in later experiments.

Design of NIR Calibration Device
A NIR calibration device is mainly composed of three parts, namely, calibration template, NIR light source, and heat dissipation system.

Selection of Calibration Template
Camera calibration algorithms are used to solve the parameters of the camera's mathematical model, which relates the actual geometric information of the calibration template to the positions of the markers imaged by the camera. Circles, triangles, rectangles, and crosswire are commonly used markers with parameters of the center, radius, vertex, and intersection, respectively.
Checkerboard and Halcon calibration templates are the most common industrial templates. Relevant literature states that the position extraction accuracy of an ellipse (the result after circular projection) can reach 1∕100 pixels, 18 and the extraction accuracy of the corner points of checkerboard templates is only at the subpixel level (1∕10 pixels). As a result, the Halcon transparent glass calibration template with the circular markers was selected for the design. The black circular markers on the calibration template exert a strong absorption effect on the NIR light wave and the glass part effectively transmits light, which can be used to obtain images with sharp contrast.

NIR Source and Heat Dissipation System
A sequence of images of the calibration template from different angles is typically required by most calibration algorithms; thus, the brightness of NIR source must be appropriate and stable. The NIR source had an aluminum plate with a 100 mm 2 surface that consists of 100 (10 × 10) NIR (850 nm) lamp beads. A high-performance 12 V DC power supply was used to provide stable luminescence to the NIR light source. A lens (108-mm outer diameter, 95-mm inner diameter, 22.5-mm thickness, and 80-mm focal length) was placed 80 mm away from the light source to improve the homogeneity by scattering light. Figure 2 shows the 3-D model of the NIR source. The heat dissipation system was designed to ensure stable and durable working characteristics of the NIR source. The pure copper plate was installed at the back of lamp beads, and it was connected to the rear heat sink by four pure copper heat conduction pipes. A fan was installed to improve heat dissipation efficiency.

Assembly
A truncated sleeve combined the NIR source system with the Halcon transparent glass calibration template, which is fixed using a threaded cover plate. The inner sleeve wall was sanded to diffuse light and improve the uniformity of light projected onto the calibration template. The NIR light source was placed behind the Halcon glass calibration template to create uniform and stable imaging. The heat dissipation module was used to reduce the temperature change of the calibration template and enable the system to work stably for a long time. Figure 3 shows the 3-D model of the NIR camera calibration device in a specific band.

Temperature Test
A FLUKE 62Mini IR thermometer was used to measure the temperature of the working calibration template. The center and the edges were measured three times every 5 min. The measurement results of continuous tests for 120 min are shown in Fig. 4. The environment temperature was ∼23°C. During the operation, the temperature change of the calibration template was extremely small (<2°C). The thermal expansion coefficient of the calibration template was 1.0 × 10 −6 ∕°C. Thus, the expansion change was less than 2.0 × 10 −5 . Compared with the common camera calibration accuracy of 10 −4 orders of magnitude, small temperature changes do not remarkably affect the calibration result.

NIR Camera
This experiment used two Basler avA1000-120 km CCD cameras, imaged in visible-light and NIR fields. The resolution (H × V) is 1024 × 1024 pixels, and pixel size is 5.5 × 5.5 μm 2 (H × V). Two Computar M1620-MPW2 with focal lengths of 16 mm are utilized as camera lenses. FS03-BP850 at the 850-nm NIR band and FS03-BP440 at the 440-nm purple light band were employed to investigate the camera's mathematical model in a specific band. Figure 5 shows the relationships between wavelength and transmittance.   Temperature versus time curve of the calibration template surface "corner 1" represents the upper left corner, "corner 2" represents the lower right corner, and "center" represents the center of the calibration template.

Extraction of Calibration Points
Given that the calibration method is unchanged, improved results can be obtained by increasing the accuracy of the calibration template and extracted points in the image. 19 In this study, the accuracy of the selected Halcon transparent glass calibration template is determined in the manufacturing process, the extraction accuracy of feature points in the image is important in optimizing the calibration accuracy.
The calibration image obtained by the NIR camera is presented in the form of overexposure to eliminate the influence of background texture, as shown in Fig. 6. Each marker has a complete structure and clear boundary.
The grayscale of the image was concentrated in the range of 0 to 20 and 200 to 255 with visible contrast. The boundary was approximated as a series of ellipses, and the gray value was almost unchanged in the background and inside the elliptic boundary. Salt-and-pepper noise was the main noise form.
According to relevant literature, the gray centroid method is most effective for extracting the center of small ellipses or circles (i.e., several to a dozen pixels) in the image. 20 According to the specific imaging situation, this study proposed a gray centroid method based on elliptic boundary.
Before extracting the center of the ellipse or circle, the recognition area requires manual determination, and the grid must be divided to ensure that each grid has only one circular marker. The method involves the following main steps.
(1) Threshold segmentation and median filtering to remove background and salt-and-pepper noise; (2) Image inversion; (3) Extraction of the image edge using the canny operator; (4) Elliptic equation is obtained by using the leastsquares method; (5) Calculation of the gray centroid for the interior of the ellipse (including edge points).
The formula can be expressed as follows: where center is the image coordinate of the center of the ellipse.
E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 7 ; 3 2 6 ; 2 9 9 x ¼ where ðx; yÞ is the image coordinate of the marker center, n pixels comprise the inside of the boundary, Iðx i ; y i Þ is the gray value of the i'th pixel in the marker of the original image, and t is the gray threshold for removing background noise. The process is shown in Fig. 7. Eighteen views of the calibration template were imaged. The marker centers in images were extracted by applying the gray centroid method based on elliptic boundary, and the reprojection errors (a geometric error that corresponds to the image distance between a projected point and a measured one) were calculated, as shown in Fig. 8. The reprojection errors of centers were less than 0.1 pixels and more accurate than those of the traditional ellipse fitting method (≤0.3 pixels). The proposed method is thereby proven to be effective in improving camera calibration accuracy.

Camera Calibration
Basler avA1000-120 km cameras were calibrated using Zhang's Camera Calibration Toolbox for MATLAB. This calibration method is widely used for visible-light cameras due to its simple operation and high accuracy. 21 Zhang's algorithm uses the checkerboard calibration template; thus, the algorithm for the feature point extraction needs to be changed. Section 4.2 presents the specific change process.
To verify the validity of the calibration device for the NIR dynamic navigator, binocular vision calibration experiments were carried out in this study.
In these experiments, NIR light source at 850 nm band and visible light source (including 440-nm purple light band) were selected for calibration. Moving the camera calibration device freely in the measurement range of the camera enabled multiple views of planar patterns at different orientations. Left and right cameras were calibrated by using the famous Zhang's method. Section 2.1 discusses the calculation of the relative position between the two cameras. Table 1 shows the partial calibration results. The accuracy of the NIR camera calibration was 2.0 × 10 −4 .
The uncertainty of the optimized result is represented as Std in this study. The pseudo-formula is expressed as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 8 ; 3 2 6 ; 1 1 5 sigma x ¼ stdðexÞ; (8) where ex is reprojection error and std represents the standard deviation operation: where D is the derivative result of projection pixel coordinates to camera calibration parameters: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 0 ; 6 3 ; 5 1 5 Std ¼ 3 A small Std value indicates high accuracy of the camera calibration.
To evaluate the advancement of proposed camera calibration methods in this study, Table 2 listed the results of several existing calibration methods. The proposed method is shown to be better than the other methods.

3-D Point Cloud Reconstruction
The binocular vision system calculated by the NIR camera calibration device was tested to verify its reliability and accuracy. The camera calibration template is also used as the calibration tool. The graphic accuracy can reach 1 μm, and a large number of feature points (41 × 41) can be collected for statistics.
The 3-D point cloud reconstruction is used to test the accuracy of calibration results and to verify the necessity of camera calibration in a specific band.
Two Basler avA1000-120km cameras were fixed on an optical bench with a baseline distance of 300 mm. Two cameras simultaneously photograph the NIR calibration device with a fixed altitude and position to obtain images. According to the extraction method described in Sec. 4.2, 39 × 39 effective extraction points were obtained (the outermost circular dots of the calibration template were excluded to improve the efficiency and accuracy of the meshing). Given that their layout of the markers on the calibration template was determined, the markers were arranged in a regular order after imaging. One-to-one correspondences of the effective extraction points on images were easily obtained, which provided convenience for subsequent 3-D reconstruction. Coordinates under the world coordinates were obtained to reconstruct the 3-D point cloud. Figure 9 shows the reconstruction results of the calibration template, where the mean square error is 0.01 mm. The 3-D reconstruction accuracy of the NIR binocular vision can reach 0.1 pixels.

Analysis of Experimental Data
Comparison of the three sets of data in Table 1 under the same camera showed that the equivalent focal lengths in the horizontal and vertical direction had obvious regular changes. The calibration value was smallest in the 440-nm purple light band, medium in the visible light band, and largest in the 850-nm NIR band. The chromatic aberration effect of lens in Sec. 2.2 states that in normal circumstances, a long working wavelength of the same lens or group of lenses results in a long equivalent focal length, which is consistent with the experimental results.
Further analysis of the experimental data in Table 1, the calibration results of visible band and 850-nm NIR band show that the changes of X_Value or Y_Value are ∼0.2% of the whole. The variable brought by camera calibration in different bands cannot be ignored. If the band spread is enlarged, then a large change is expected. Furthermore, NIR dynamic navigator working in a specific band must be calibrated in the same band to obtain realistic camera parameters. Through a longitudinal comparison of X_Value and Y_Value, the influence of wavelength is basically the same in the horizontal and vertical directions. Table 2 Comparison of results obtained from different calibration methods "proposed" represents the method proposed in this paper, "proposed * " represents the calibration results by using the NIR camera calibration device and ellipse fitting method to extract the feature points.

Method
X_Value  Table 2 shows that the proposed camera calibration has improved accuracy of results by an order of magnitude compared with those of existing methods. Furthermore, feature point extraction methods also notably influence accuracy. Comparison of the results using different feature point extraction methods leads to the conclusion that the gray centroid method based on elliptic boundary can further improve calibration accuracy. These findings present strong pieces of evidence that the proposed method improves the accuracy of NIR camera calibration.
After reconstructing the 3-D point cloud, the coordinates of the feature points in the world coordinate system were obtained. The feature point spacing on the calibration template was accurate and known, which was used as the standard value. The 3-D reconstruction feature point spacing was selected as the measured value to verify the accuracy of 3-D reconstruction. The calibration template used in this calibration had the spacing of 2.0000 mm between feature points. To prevent repetition, the distances between a point and its neighbors on the left and below were calculated, amounting to 2964 data. A statistical histogram was established and the curve was fitted with a normal distribution. The mean value was 2.0001 mm, which can approximately be the true value, and the mean square error was 0.0107 mm. Figure 10(a) shows the specific result. The NIR camera calibration device can thus be considered to effectively calibrate the NIR dynamic navigator.
Camera parameters under visible light were used to reconstruct the 3-D point cloud of the images obtained in NIR and to detect the feasibility of the traditional scheme. The process was similar to the 3-D reconstruction in NIR, and the results are shown in Fig. 10(b). The statistical histogram of adjacent intervals conformed to the normal distribution. The mean value was ∼1.9963 mm, which was 0.0037 mm different from the truth value of 2.0000 mm. There is ∼0.2% systematic error. Systematic errors occur if the visible light calibration device is used for the visual measurement system working in NIR; thus, the NIR dynamic navigator must be calibrated in the specific band. Figure 10(d) shows the standard deviations of two reconstruction results. No substantial difference in accuracy is observed between the results of camera calibration under NIR and visible light.

Conclusion
Existing methods of NIR camera calibration produce inaccurate results, which, in turn, affect the accuracy of the NIR dynamic navigator. Thus, this study designed a calibration device for the NIR camera in a specific band. With the help of the gray centroid method based on elliptic boundary to extract feature points, the accuracy of the NIR camera calibration is improved to the accuracy range (<0.02%), which is crucial to the application of NIR cameras in visual measurement. The 3-D reconstruction accuracy of NIR binocular vision can reach 0.1 pixels. The proposed calibration method in this study can be used not only for NIR bands but also for each band by replacing the backlight source system. It is a reliable camera calibration method for visual measurement. Ming Gong is a professor-level senior engineer at the Experiment Center of Engineering and Material Science, University of Science and Technology of China, mainly engaged in the research of mechanical properties of shape memory alloy.