The recent decade has seen tremendous growth in the machine vision applications as they become the new driving forces in precision engineering. Many of these applications highly benefit from the improvements of camera calibration techniques, which primarily fall into two categories: photogrammetry calibration¹ and self-calibration.² Compared with the latter which does not use any specific calibration target, the former usually provides higher accuracy by using a precisely-constructed calibration target and is therefore the interest of this letter. A representative photogrammetry approach is Zhang's method, which uses a planar pattern as the calibration target and is well known for its flexibility, robustness, and low cost.¹ This method and its extensions implemented in the OpenCV library are considered as the conventional technique in this letter.

The literature in the relevant field addresses two major sources of error that affect the camera calibration results. The first one is the imperfection of the calibration target. Since the assumptions made for the conventional camera calibration are based on a perfect planar target with ideal patterns, the imprecision of the calibration target may lead to inaccurate results.³ The second problem is the uncertainty in locating the control points directly from the geometries of the calibration patterns in the captured raw images which suffers from lens distortion as well as perspective distortion.⁴

This letter proposes an advanced geometric camera calibration approach to overcome the aforementioned two problems without loss of the original advantages of the conventional technique. The proposed technique uses a sophisticated lens distortion model that takes the radial, tangential, and prism distortion into account, and achieves a precise localization of the control points with a novel refinement process using a frontal image concept and an advanced digital image correlation (DIC) scheme; in addition, the imperfection of the calibration target can be compensated.

For the ideal pinhole model, assuming that the world coordinate is placed on the calibration target with its surface as the XY plane, the relation between the three-dimensional world coordinate of a calibration target point [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm M} = [X_w, Y_w, Z_w]^T$\end{document} $\mathit{M}={[X_w,Y_w,Z_w]}^T$ and its corresponding location [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm m} = [u,v]^T$\end{document} $\mathit{m}={[u,v]}^T$ in the image plane can be expressed as:

where s is a scale factor; [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm A}$\end{document} $\mathit{A}$ is the intrinsic matrix, with α and β the horizontal and vertical focal length in pixel unit, γ the skew factor, and (u₀, v₀) the coordinates of the principal point; [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm R}$\end{document} $\mathit{R}$ and [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm T}$\end{document} $\mathit{T}$ are the extrinsic parameters that denote the rotation and translation relating the world coordinate system to the camera coordinate system. Due to nonlinear optical distortion, the above equation is not sufficient for accurate camera calibration. In spite of the fact that some very complex models exist,⁵ in practice they induce more instability rather than accuracy because of the high order distortion components. Here, the lens distortion is compensated by: where (a₀, a₁, a₂), (s₀, s₁), and (p₀, p₁) represent the radial, prism, and tangential distortion coefficients, respectively, (u, v) denotes the distortion-free pixel location, and (u^′, v^′) is the corresponding distorted point. To avoid calculation instability during the optimization process, in Eq. 2, (u, v) and (u^′, v^′) are usually converted to their normalized terms (x_cn, y_cn) and [TeX:] \documentclass[12pt]{minimal}\begin{document}$(x^{\prime }_{cn},y^{\prime }_{cn})$\end{document} $(x_{cn}^{\prime },y_{cn}^{\prime })$ by applying the intrinsic matrix [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm A}$\end{document} $\mathit{A}$ .

Given the locations of the control points in the world coordinate as [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm M_{\bm i}}$\end{document} ${\mathit{M}}_{\mathit{i}}$ and in the image plane as [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm m_{\bm {ij}}}$\end{document} ${\mathit{m}}_{\mathit{ij}}$ , the calibration process involves a nonlinear optimization with the cost function defined as:

where k and l denote the number of control points in the calibration target and the number of images, respectively, [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm \varphi } = (a_0, a_1, a_2, s_0, s_1, p_0, p_1)$\end{document} $\mathit{\varphi }=(a_0,a_1,a_2,s_0,s_1,p_0,p_1)$ , P denotes the projection of control points onto the image planes according to Eqs. 1, 2, and [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm r}$\end{document} $\mathit{r}$ is the Rodrigues vector presentation of [TeX:] \documentclass[12pt]{minimal}\begin{document}${\bm R}$\end{document} $\mathit{R}$ .^{1, 6} The optimization is performed using the Levenberg–Marquardt (LM) algorithm.

The method described above relies on the intrinsic assumption that the control points on the calibration target are perfectly positioned. In reality, printing imprecision during calibration pattern fabrication is inevitable, especially for low-cost hardware systems. To deal with this issue, the proposed technique allows the world coordinates of the control points to be unknown, and they will be accurately determined together with the camera parameters. To ensure the uniqueness of the calibration target during optimization, a geometric constraint on three noncollinear control points, named markers, is applied.⁶ The constraint enforces planarity of the markers by setting their Z world coordinates to zero, and requires the distance between any two of the three markers to be accurately measured to solve for the scale factor. The markers, normally made at the corners of the calibration panel, also help determine the orientation of the panel.

An accurate ellipse fitting technique can be employed to detect the centers of the calibration circles.⁷ However, as the control points detected in the raw images suffer from lens and perspective distortion, their true locations cannot be accurately determined, and this will further lead to calibration errors. To cope with this issue, a two-step refinement with the frontal image concept, which is free from lens and perspective distortion, is employed. The refinement is conducted after the camera parameters are coarsely determined using the method described above. The raw images are successively undistorted by applying Eq. 2 and then reversely projected to the world coordinate system through using Eq. 1. The frontal image is generated from a direct scaling of the world coordinates. It is noteworthy that while the conventional subpixel edge and circle fitting method can be used to detect the control points in the frontal images, the method is not recommended because the gradient calculation involved in edge detection is sensitive to noise and the circle approximation does not yield high accuracy. On the other hand, as illustrated in Fig. 1, the transformation of each image from each location of the captured calibration board to the frontal image allows using the DIC concept to accurately locate the position of each control point by comparing the frontal image with the synthesized templates. Datta ⁴ achieved subpixel-location detection of the control points by carrying out a quadratic fitting in the neighborhood regions based on their correlation coefficients, but such a peak-finding approach is less accurate than the iterative schemes.⁸ Here, the proposed technique employs a high-accuracy cross-correlation algorithm, and the correlation coefficient function is:^{6, 8}

where a is a scale factor, b is an intensity offset, and f(x_i, y_i) and [TeX:] \documentclass[12pt]{minimal}\begin{document}$g(x^{\prime }_i,y^{\prime }_i)$\end{document} $g(x_i^{\prime },y_i^{\prime })$ denote the intensity values at the i–th pixel in the template and frontal images, respectively. The template pattern is a square subimage of N pixels with its center as the center of the circular target point. Denoting the subimage center as (x₀, y₀) and its shift amount between two subimages as (u, v), the correlation shape function considering both translation and rotation is where u_x, u_y, v_x, and v_y are coefficients of the shape function. To determine the eight unknowns (u, v, u_x, u_y, v_x, v_y, a, b), the Newton–Raphson algorithm is employed to minimize C in Eq. 4. With the detected (u, v), the location of each control point in the frontal image can be directly determined. Then, these points in the frontal image are reversely projected back to the image plane to achieve hyperaccurate localization of the control points.

The rationale of the proposed method is that although the imprecise control points detected in the raw images lead to inaccurate camera calibration parameters, they offer good initial information about the relation between the camera and each scene which can be further processed to achieve very accurate localization of the control points. This information helps detect the camera intrinsic and extrinsic parameters as well as the world coordinates of the control points with high accuracies. The procedure of the proposed technique is summarized as follows:

To demonstrate the validity of the proposed technique, computer simulation along with real experiment have been conducted. The results suggest that no less than 40 control points and 15 positions are required to ensure a reliable calibration. Both simulation and experiment presented below use a flat panel with 10 × 7 ring patterns whose grid distance is 25.4 mm (as illustrated in Fig. 1), and 20 images at different positions are used in the calibration.

In the simulation, the images are synthesized with camera parameters obtained from a real calibration where the radial, tangential, and prism lens distortion are considered. In addition, Gaussian noise with a standard deviation of 0.2% of the 25.4 mm distance is added to the position of each ring pattern in the x and y directions, and the images are also blurred and Gaussian noise with σ = 2.0 is added. Figure 2 shows the errors in locating the positions of the control points against their true values. Using Heikkila's technique⁷ to detect the control points directly in the raw images, the effect of perspective distortion where the ellipse center is not the actual circle center is obvious. In contrast, with the frontal image and DIC concepts, the distortion effect is removed, and over 20 times localization improvement in terms of the root-mean-square error (RMSE) in both x and y directions is achieved. The residual of the calibration, named reprojection error and defined as the RMSE between the projection of the control points to the image planes and their measured locations, is 0.00086 pixels. Table 1 shows the results of the intrinsic parameters. It can be seen that the camera parameters can be accurately retrived once the control points are detected with high precision.

Table 1

Intrinsic calibration parameters.

For the real experiment, Fig. 3 shows the displacement error vectors between the detected and projected positions at each control point of the calibration target. With the conventional method and the proposed one, the reprojection errors are 0.3848 and 0.0098 pixels, respectively. Because of the insufficiency of the lens distortion model and the imperfection of the calibration target, the reprojection error is noticeably polarized with the conventional method in spite of its control refinement. With the proposed method, the error behaves more isotropic, indicating that the lens distortion and the imprecision of the calibration target have been successfully compensated.