2.1

Single Camera Calibration

To start the camera calibration, a camera model has to be chosen and be described. The OpenCV calibration algorithm utilizes a pinhole camera model and introduces radial and tangential distortion [3]. Distortion correction is vital since the presented multi-camera system uses low-cost board-level cameras with S-mount lenses, which introduce an amount of distortion to the images that can not be neglected.

At first a transformation from a three-dimensional point P_w(X_w, Y_w, Z_w) in world coordinates to a point P(u, v) in image pixel plane coordinate system has to be found (see Figure 3). The equation (1) transforms a point P_w to a point P_c(X_c, Y_c, Z_c) in the camera coordinate system, where R is a 3x3 rotation matrix and t is a 3x1 translation vector.

Figure 3.

Coordinate system for camera calibration

The point P_c is now projected through the pinhole model in order to obtain physical coordinates on the image plane as P(x, y), see (2).

By introducing radial distortion coefficients (k₁, k₂, k₃), the corrected point coordinates P_k(x_k, y_k) are defined as following:

With k₁, k₂ and k₃ being the radial distortion coefficients and r² = x² + y². Due to a lens not aligned perfectly to the image plane, tangential distortion is introduced. Its correction for a point P_p(x_p, y_p) can be described as:

In summary the distortion coefficients are defined as d = (k₁, k₂, p₁, p₂, k₃). The tangential and radial distortion corrected point P_q(x_q, y_q) is, as combination of the equations (3) and (4), defined as:

To translate the physical image plane coordinates to image plane pixel coordinates, the camera matrix A is needed, which contains information about the focal length in mm as f_x and f_y, as well as the optical center in pixel coordinates as (c_x, c_y). γ represents the skewness, which is the angle error in between the two axis of the pixel array. For industrial grade image sensors skewness is usually small and can be neglected.

In conclusion a point P(u, v) in pixel coordinates is defined as:

2.2

Stereo Camera Calibration

Stereo camera calibration, or binocular calibration, will, in addition to the already obtained intrinsic and extrinsic parameters, get the relative position of one camera to the other of a stereo pair along with matrices necessary for the reconstruction of the scene.

We suppose the left camera extrinsic parameters are rotation matrix R_L and translation vector t_L. For the right camera these are R_R and t_r. From these matrices and vectors the translation vector from left to right camera coordinates t_LR and the corresponding rotation matrix R_LR are derived.

The keyword for stereoscopic reconstruction is epipolar geometry, which encapsulates the relation of the projective geometry between two views. The centrepiece of this is the fundamental matrix F, which derives from the relation given in equation (8), with x as the projection of a real-world point X in the left camera and x′ in the right camera frame. The equation could be solved for image points with known real world distances obtained from both cameras of a stereo pair. For a more in-depth explanation of epipolar geometry refer to [4].

To be able to transform images pairwise abiding to epipolar geometry, rectification homographies H_L and H_R need to be found. With known fundamental matrix F, the algorithm described in [5] can be utilized.

Figure 6 shows frames of pair 2 distortion corrected and rectified, which means the rectification homo-graphies are applied.

To be able to reproject image points to 3D coordinate space, disparity has to be introduced. Figure 4 depicts a simplified projection of a point P onto two already rectified images. Since these images are rectified, the projections P₁ and P₂ are located on one epipolar line. Therefore v₁ = v₂ and u₁ ≠ u₂. The equation for disparity is given by (9) with B as baseline or distance between the optical centers of the cameras (C₁ and C₂), f as focal length and z as distance to point P in 3D world coordinate system.

Figure 4.

Left side: point P projected onto rectified frames of stereo pair; right side: point P projected onto rectified frames of stereo pair in top view.

One step in stereo calibration is still missing, the computation of the disparity-to-depth mapping matrix Q. Its definition is given by equation (10). Since the vectors are noted in homogeneous coordinates their fourth entry refers to scale. With this equation a disparity map can be reprojected to the world coordinate system in 3D (see section 3).

2.3

Calibration Procedure

In order to apply the above elucidated relations, a set of points with known distances is needed. Therefore a chessboard type calibration target is used, as shown in Figure 5. A set of images with different positions of the target are taken, where the target should be moved to every position in the measurement volume in order to get a strongly calibrated stereo vision system. The position of the corners on the calibration target are detected by using the OpenCV function findChessboardCorners() and, to get the corner position with subpixel precision, cornerSubPix() is called afterwards. This routine is carried out for every image during the calibration process. This leads to a set of points for every camera where every point refers to a corner on the chessboard pattern. With these points and an array with distance values in mm corresponding to the dimension of the rectangles in the calibration pattern, the cameras can be calibrated. For every stereo pair at first the extrinsic and intrinsic parameters of the two partaking cameras are calculated (see section 2.1) using the OpenCV function calibrateCamera(), which is based on [6] and [7]. This routine delivers the camera matrices A_L and A_R, distortion coefficients d_L and d_R, rotation matrix and translation vector for every pattern view and a reprojection error. The distortion coefficients are later used to undistort images, as depicted in Figure 6 on the left side.

Figure 5.

Frames of camera 1 to 4 (left to right) with chessboard calibration pattern. The image frames are uncorrected, neither undistortion nor rectification transformations are applied.

From the rotation and translation matrices for every pattern view, the rotation matrix R_LR and translation vector t_LR from left to right camera are derived.

The above mentioned chessboard corner points in image pixel coordinates are now undistorted using the function undistortPoints(), and further on used to obtain the F matrix via the OpenCV function findFundamentalMat(). This function utilizes the random sample consensus (RANSAC) algorithm to solve the problem delineated in equation (8). With the obtained F matrix the above mentioned rectification homographies can be calculated. In OpenCV the function stereoRectifyUncalibrated() computes the rectification homographies H_L and H_R using the algorithm presented in [5]. On the right side Figure 6 depicts rectified images of stereo pair 2.

Figure 6.

Images obtained from stereo pair 2 (left: camera 1, right: camera 3). The image pair shown on the left side is distortion correct, the pair on the right side is rectified. On the greyscale pattern two objects (one cube and one cone) are placed, see yellow markers in left image.

After rectification a disparity map can be calculated using a matching algorithm, for example semi global block matching (SGBM), this procedure is explained in section 3. As mentioned above, the Q matrix is needed to reproject a disparity map to 3D. The connection between disparity, world points and Q is given in (10). Since we are using the OpenCV function stereoRectifyUncalibrated(), which does not return a Q matrix, the disparity-to-depth mapping matrix needs to be calculated in another way. Therefore the equation (10) is utilized. At first the points obtained from the calibration target are triangulated using OpenCVs triangulatePoints(), which results in a matrix with points in homogeneous coordinates, notated as B in (11). The matrix D in the same equation is acquired by applying H_L, respectively H_R, to image points obtained by findChessboardCorners() and cornersSubPix(). Equation (12) describes the derivation of a pseudoinverse matrix, which is in (13) used to convert equation (11) to obtain Q. In this way it is possible to calculate the disparity-to-depth mapping matrix based on the points retrieved from the different views of the calibration target. Assuming the calibration target was moved through the whole observed volume, maximum and minimum disparity values can also be calculated using these points.

3.1

Constructing Disparity

The OpenCV implementation of the semi-global block matching (SGBM) algorithm is applied to the rectified images of stereo pair 2 (see Figure 6, right side). The OpenCV class StereoSGBM uses a modified algorithm from [8], whereas, instead of mutual information cost function, a simpler sub-pixel metric from [9] is implemented.

In the first step pixelwise cost calculation is done, the cost is calculated as the absolute minimum difference of intensities in the range of a half pixel in 5 directions along the epipolar line. As the name of the algorithm suggests, a global smoothness constraint is approximated. The smoothed cost for a pixel (or block, depending on the chosen block size) and disparity is calculated by summing the costs of all minimum cost paths that end in the pixel or block. The disparity map is determined by selecting a disparity with corresponding minimum cost for every pixel of the source image. This process is done two times, the first time from left to right image, the second time from right to left image. Therefore a left and right matcher instance is created. The input minimum disparity value is obtained from the matrix D in equation (11), as well as the number of disparities, which is the maximum disparity value minus the minimum disparity value. From these two maps one combined disparity map is preserved using the Weighted Least Squares filter with a left-right-consistency-based confidence map. In OpenCV these functions are implemented in the ximgproc.DisparityWLSFilter class. Figure 7 depicts the output disparity maps.

Figure 7.

Disparity map of pair 2, corresponding to image pairs shown in Figure 6. From left to right these images are left-right disparity map, right-left disparity map and the filtered disparity map. The disparity is mapped to values ranging from 0 to 255, hence dark pixel represent areas further away from the camera pair, brighter pixel represent areas closer to the camera pair (except for the right-left matched map). In order for the SGBM algorithm to work properly, the surface has to be structured or a structure has to be projected. In these frames this is not implemented, hence the only valid section is the structured area (see Figure 6).

3.2

Constructing Point Clouds

The obtained disparity images (see Figure 7) need to be reprojected to 3D world coordinates. Therefore the disparity-to-depth mapping matrix Q is calculated, see section 2.2. In OpenCV the function reprojectImageTo3D() transforms a disparity map to a 3-channel point cloud. For each pixel (u, v) and its corresponding disparity value disparity(u, v) the equation (10) is solved. This results in a dense point cloud with 5.038.848 points. Figure 8 depicts the valid part of the resulting point cloud.

Figure 8.

Two views of the point cloud resulting from the reprojection of the disparity map to 3D world coordinates. Only the valid area (cf. Figure 7) is depicted.