Epipolar constraint of single-camera mirror binocular stereo vision systems

. Virtual binocular sensors, composed of a camera and catoptric mirrors, have become popular among machine vision researchers, owing to their high flexibility and compactness. Usually, the tested target is projected onto a camera at different reflection times, and feature matching is performed using one image. To establish the geometric principles of the feature-matching process of a mirror binocular stereo vision system, we proposed a single-camera model with the epipolar constraint for matching the mirrored features. The constraint between the image coordinates of the real target and its mirror reflection is determined, which can be used to eliminate nonmatching points in the feature-matching process of a mirror binocular system. To validate the epi-polar constraint model and to evaluate its performance in practical applications, we performed realistic matching experiments and analysis using a mirror binocular stereo vision system. Our results demonstrate the feasibility of the proposed model, suggesting a method for considerable improvement of efficacy of the process for matching mirrored features. © The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI. [DOI: 10.1117/1.OE.56.8.084103]


Introduction
2][3] Traditionally, a binocular stereo vision system is composed of two cameras or a moving camera, capturing the object's images from different directions.However, sensors that are built using two cameras are characterized by large size and poor flexibility, while those that utilize one camera lack instantaneity and synchronization.][6] Compared with conventional two-camera vision systems, a virtual binocular stereo vision system is characterized by good synchronization, compact structure, low cost, and high flexibility.However, in the applications that use such virtual binocular stereo vision systems for 3-D measurements, the two pivotal tasks, calibration of the system and feature matching, are different from those used in traditional two-camera systems.
When using a two-camera stereo vision system, calibration refers to the process of determining the intrinsic parameters of the two cameras and their structural parameters. 7hen, feature matching is performed, and the feature points are effectively constrained using the conventional model with the epipolar constraint. 8However, for a single-camera virtual binocular vision system, each captured image is separated into two parts, which are projected by the real target and its mirror reflection.The usual approach is to separate the two parts of the image, creating a binocular system with two virtual cameras.Therefore, the calibration task of such a system consists of determining the intrinsic parameters of the single camera and the structural parameters of the two virtual cameras.Based on the above interpretation of a virtual binocular system, a two-step calibration method (TSCM), 9,10 consists of the following steps.First, the catoptric mirrors are removed, using only the single camera to capture the calibration target images and to calculate the camera's intrinsic parameters.Then, the catoptric mirrors are reincorporated into the system and one calibration target image is recaptured to determine the system's structural parameters.
2][13] However, previous works mainly aimed at solving the problem for the condition in which the same target is projected onto the single camera at the same reflection times.5][16][17] Thus, the two real cameras are replaced by virtual cameras formed by a single camera and mirrors. 18or a single-camera mirror system, the region of interest (ROI) usually covers the entire image, which significantly increases the matching error.To establish the geometric principles of the feature-matching process of a single-camera mirror binocular stereo vision system, we derived the epipolar constraint between two image parts for a single-camera model.This model combines the traditional epipolar constraint and the particularity of a single-camera mirror binocular stereo vision system, providing the constraint between the image coordinates of the real target and its mirror reflection.

Review of the Traditional Epipolar Constraint
Model Epipolar geometry is convenient for describing and analyzing multicamera vision systems.It is used to represent the geometric relationship between two viewpoints of the same scene based on a few corresponding points in a pair of images.This relationship, which is formulated as a matrix (called the fundamental matrix), can further be used for simplifying the ROI, computing the displacements between cameras, and rectifying the stereo image pairs.

Camera Model
Image formation in a camera can be described by a widely used pin-hole model. 19The coordinates of a 3-D point P ¼ ½x; y; z T in the world coordinate system and its image plane coordinate p ¼ ½u; v T are related through E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 6 3 ; 6 2 6 s where s is a scale factor and M is a 3 × 4 matrix, called the perspective projection matrix.Because the homogeneous coordinates of a vector m can be written as m ¼ ½m T ; 1 T , we obtain sx ¼ M X.Thus, the perspective projection matrix M can be written as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 6 3 ; 5 0 6 M ¼ A½ R t; (2 where A is a 3 × matrix mapping the camera coordinate system to the image coordinate system, and ½ R t are the structure coefficients of the two-camera vision system (rotation matrix and translation vector, respectively), which transform the world coordinate system to the camera coordinate system.Note that the matrix A depends only on the system's intrinsic parameters, which capture the optical, geometric, and digital properties of the camera, while the 3 × 4 matrix ½ R t contains only the extrinsic parameters, which describe the transformation between the two coordinate systems.

Two-Camera Epipolar Constraint Model
Epipolar constraint is one of the most important principles in the binocular stereo vision, and is also a fundamental constraint underlying all self-calibration techniques. 20Consider a two-camera stereo vision system shown in Fig. 1.Note that P is a 3-D point; p l and p r are its projections onto image I l and image I r , respectively; C l and C r are the optical centers of the left and right cameras, respectively.The plane π, defined by the three spatial points P, C l , and C r , is known as the epipolar plane.The intersection of the epipolar plane π with the image I r is termed as the epipolar line, and is denoted by l pr .Thus, the corresponding point p l in the image plane I l of p r must be constrained to the line l pl .This model can also be described by geometric deduction, as shown in Fig. 1.Define K l and K r as the intrinsic matrices of the two cameras, respectively, and let ½ R t denote the transformation between the coordinate systems of the two cameras.Under the pin-hole model, the following equation holds: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 3 2 6 ; 6 5 3 Here, ½t × is an antisymmetric matrix defined by the translation vector t.To simplify the above equation, define matrix F as ; t e m p : i n t r a l i n k -; e 0 0 4 ; 3 2 6 ; 5 8 8 where F is known as the fundamental matrix of the two cameras.Thus, Eq. ( 3) can be reduced to a simpler expression ; t e m p : i n t r a l i n k -; e 0 0 5 ; 3 2 6 ; 5 3 4 Geometrically, the factor F pl defines the epipolar line of point p l in the right image.Equation ( 5) prescribes that for point p l the corresponding point in the right image is located on the corresponding epipolar line.Transposing Eq. ( 5) yields a symmetric relation linking the right image to the left image.The fundamental matrix F is of great significance for camera calibration and feature matching, because it is the only geometric constraint available for two uncalibrated images.Once corresponding pairs of points for the two images are obtained, the intrinsic matrices K l , K r , and the structure coefficients ½ R t can be determined.

Single-Camera Mirror Epipolar Geometry
The epipolar geometry of a two-mirror system was first investigated by Gluckman and Nayar. 21They showed that the number of free parameters in the fundamental matrix can be reduced from 7 to 6 for a two-mirror system with no constraint on the locations of mirrors.In what follows, we develop a precise description of the epipolar constraint model of a single-camera binocular vision system.

Single-Camera Mirror Binocular System
There are two types of mirror binocular stereo vision systems.For systems in the first category, the tested target is  imaged using one real space path and one reflection path. 16hese types of binocular stereo vision systems with one real image are shown in Fig. 2.There are two images of the point P, corresponding to the two different paths.For the mirror point P 0 , the image is captured after one reflection.But for the real point P, the image is captured directly.Thus, this binocular vision device is equivalent to a device in which a real camera and a virtual camera are at fixed mirror symmetric positions.
For systems in the second category, the tested target is projected onto a single camera via two different reflection paths. 17In this system, the tested target is projected onto a single camera after one or two reflections, as shown in Fig. 3. Imaging of the target in the field of view (FOV) can be separated into two reflection paths.Using the upper slope mirror, the target can be captured after one reflection.On the other hand, two reflections are needed for imaging performed using the lower mirror.Therefore, for this virtual binocular structure, the left and right virtual cameras and images exhibit a mirror relationship.Figure 3(a) shows two different paths from the target to the camera.As a four-side symmetric system, four pairs of target images from different directions can be captured simultaneously, as shown in Fig. 3(b).For each pair, binocular images are mirror-symmetric, as shown in Fig. 3(c).

Single-Camera Mirror Epipolar Constraint Model
As is well known, in the traditional two-camera stereo reconstruction process, feature matching can be effectively performed using the epipolar constraint. 22However, for these mirror images, the epipolar constraint model exhibits different characteristics in comparison to traditional tworeal-camera systems, because the same target point is captured from different paths by a single camera and forms two image points, as shown in Fig. 4.
Owing to the unicity of the real camera and the symmetry of the real camera and its reflection, two virtual epipolar points e l and e r should coincide in the single image.In addition, the two epipolar points and two virtual target points p l and p r should be coplanar.Thus, in the real image plane, the two epipolar points and two target points should be collinear, and the two epipolar points should have the same positions.In this approach, the analysis is started from the target point P and its symmetric point P 0 .According to Eq. ( 1), the perspective projection of the two points can be expressed as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 3 2 6 ; 5 8 5 s pr As stated in the previous section, the 3-D point P ¼ ½x; y; z T can be denoted by its homogeneous coordinates P ¼ ½P T ; 1 T .Thus, the three relationships can be written as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 7 ; 3 2 6 ; 4 9 6 8 < : where ½ E t is the imaging transformation caused by the reflecting mirror, and E is the 3 × 3 identity matrix.Thus, Eq. ( 6) can be transformed as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 8 ; 3 2 6 ; 4 0 4 To simplify this model of imaging, the same element M 1 P þ m 1 can be eliminated from the above two equations, yielding the following expression for pr and pl E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 9 ; 3 2 6 ; 3 2 6 Considering the above equation from a purely geometrical perspective, the expression pl − pr describes the vector p r p l in the image's coordinate system, and t is equivalent  to the vector PP 0 in the camera's coordinate system.Thus, Eq. ( 9) can be interpreted as follows.The two-dimensional (2-D) vector p r p l is the projection of the 3-D vector PP 0 from the camera's coordinate system to the image's coordinate system.Because C r is the projection center from any viewed point including P, the principle shown in Eq. ( 9) can be used to derive another relationship among C r , C l , and e E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 0 ; 6 3 ; 5 0 7 sð pr − ẽÞ ¼ M 1 t 0 ; ( where t 0 denotes the vector C r C l in the camera's coordinate system.We assume that u is the identity vector normal to the mirror's plane.Equations ( 9) and ( 10) can be further expressed as follows: Transposing the constants a and b, and eliminating the same element M 1 u from the above equations, we obtain a relation linking ẽ, pr , and pl E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 2 ; 6 3 ; 3 5 4 1∕að pl − pr Þ ¼ 1∕bð pr − ẽÞ: According to the above equation, it can be derived that the three image points p r , p l , and e are on the same straight line in the image's plane.Because of the unicity of the real camera and the symmetry between the virtual camera and the real one, two virtual epipolar points e l and e r coincide at point e.In addition, the two epipolar points and two target points p l and p r are coplanar.Thus, in the real single image plane, the two epipolar points and the two target points are collinear, and the two epipolar points are at the same position, as shown in Fig. 5.

Single-Camera Multimirror Epipolar Constraint Model
Here, we introduce a system that is different from the one described in the previous section.In this system, the space points in the FOV can project on the single camera through one or two mirrors.As before, the analysis starts from the target point P and its symmetric points P o , P l , and P r that are reflected by different mirrors, as shown in Fig. 6.Here, P o is the virtual point reflected by mirror M 2 , and P l is the virtual point reflected by mirror M 1 .P r is the symmetric point of P o for mirror M 3 .The symmetric virtual cameras R and L of the real camera C are formatted according to the same principle.The virtual cameras R and L make the system equivalent to a binocular system.The virtual cameras L and O are the reflections of the real camera C in mirrors M 1 and M 3 , which are formatted by one reflection.The virtual camera R is symmetric with the virtual camera O about mirror M 2 , which is formatted by two reflections from the real camera C. According to Eq. ( 12), the space point P ¼ ½x; y; z T projects to the real camera C and the virtual camera L and formats two image points p and p l , which conform to the following relationship: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 3 ; 3 2 6 ; 3 1 1 1∕a l ð pl − pÞ ¼ 1∕b l ð p − ẽÞ: The same relationships exist for the real camera C and the virtual camera O, and the virtual camera R is symmetric with the virtual camera O about mirror M 3 .Thus, the following equations can be derived: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 5 ; 3 2 6 ; 2 0 6 The element po can be eliminated by combining Eqs. ( 13)- (15).Thus, the relationship between two image points in the virtual cameras R and L can be derived as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 6 ; 3 2 6 ; where ẽlr are the homogeneous coordinates of individual epipolar points in the real image, and e o , e r , and e denote the epipolar points determined by different virtual binocular structures.According to the above equations, the three image points p r , p l , and e lr can be shown to be on the same straight line in the real image's plane, similar to the system of one mirror, as shown in Fig. 7.

Experiments
Aiming at evaluating the performance of the proposed epipolar constraint model in practical applications, real experiments and analyses were performed using a mirror virtual binocular stereo vision system.Before performing the experiments and analyses, the virtual binocular vision system was calibrated.Thus, the coordinates of the epipolar point and the principal point, the distortion coefficients, and the focal length of the camera lens were obtained.In future experiments, the results of this calibration will be used in a direct manner.

System Calibration
The experimental system was established according to the one-mirror system described in the previous section. 16he experimental system included a camera and a reflecting mirror, which were fixed on the experimental platform.The camera was IMPERX-IGV-B1601M version, with the frame frequency 15 fps, resolution of 1624 × 1236 pixels, focal length of the lens at 8.5 mm, and size of the charge coupled device at 2/3 in.The setup is shown in Fig. 8.
To calibrate the system precisely, we used the TSCM 9 and Zhou et al.'s method in Ref. 6.For the former method, the system's intrinsic parameters and structural parameters were calibrated separately by removing the mirror and fixing it, while for the later one, these parameters were calibrated without removing the mirror.The calibration target used in this process was a ceramic plane with circular features organized in a 7 × 7 array; the diameters of the dots and the center-to-center distances between adjacent dots were 4 and 8 mm, respectively.The calibration plane images captured for calibrating the system's intrinsic parameters and structural parameters are shown in Fig. 9.
In the calibration process using the TSCM, eight calibration plane images were used for the extraction of intrinsic parameters; two of them are shown in Figs.9(a) and 9(b).Calibration of structural parameters requires only one image, which includes the real calibration plane image and its mirror reflection, as shown in Fig. 9(c).However, in the calibration process using Zhou et al.'s method, four mirror images of the calibration plane were captured for the extraction of intrinsic parameters, which included a total of eight calibration plane subimages, as shown in Fig. 10.Calibration of structural parameters requires only one image, which is the same as TSCM.The results of the two calibration procedures are listed in Table 1.
Here, f x and f y denote the focal lengths of the lens in the x and y directions; ðu 0 ; v 0 Þ are the image coordinates of the principal point; k 1 and k 2 represent the two-order lens distortion coefficients; R and T are the rotation matrix and the translation vector, respectively, of the virtual binocular structure; E RP denotes the reprojection error; and ðx e ; y e Þ are the coordinates of the single epipolar point.The structural parameters were calibrated using the same image and have the same rotation matrix R and transition vector T. For a single camera vision system, the reprojection error is a significant parameter showing the mapping accuracy from the 3-D space to 2-D images.According to the calibration results, the TSCM achieved higher reprojection accuracy compared to Zhou et al.'s method.Thus, in the following feature-matching experiment, the TSCM calibration results will be used.

Validation Experiment
The experimental setup has been precisely calibrated using the TSCM, as described in the previous section.To validate the proposed single-image mirror epipolar constraint model, a feature-matching experiment of the calibration plane based on the single image was performed.The validated process is shown in Fig. 11.First, an image of the calibration plane was captured by the calibrated experimental setup for testing, as shown in Fig. 11(a).The calibration plane that was used here was the same as that used in the calibration experiment.Then, we extracted the coordinates of all of the 2 × 49 feature points of the tested image, using the ellipse fitting method. 23The results of this extraction, after correcting for the image distortion, are shown in Fig. 11(b).Next, we established the relation between the two arrays of coordinates of feature points in a point-by-point manner, according to the proposed epipolar constraint model, as shown in Fig. 11(c).Finally, by rebuilding all the 49 feature points according to the stereo vision model and previous calibration results, we obtained the 3-D space point coordinates.For a more scrupulous validation of the proposed mirror epipolar constraint model, three additional experiments were performed according to the same process as the first experiment.The four reconstruction results are shown in Fig. 11(d).
The error analysis of the four experiments was performed with the aim of precisely evaluating the errors between real feature and reconstructed points.We calculated coordinates of all points in the real camera coordinate system and analyzed the offset distances between these points and the real calibration plane by comparing the measured distance of the adjacent feature point with the real value of 8 mm.The     2. It can be seen that the average absolute and relative errors are 0.05 mm and 0.6%, respectively.

Real Feature-Matching Experiment
To validate the proposed epipolar constraint rule on practical applications, a real feature-matching experiment was performed and is reported in this section.For comparison, we also report the results of feature matching performed without the epipolar constraint.The real target image captured by the experimental system is shown in Fig. 12(a).Feature extraction was performed using the oriented FAST and rotated BRIEF (ORB) method, 24 and the results of matching obtained without the constraint rule are shown in Fig. 12(b).Apparently, the matching errors are very high.The results obtained using the proposed epipolar constraint model are shown in Fig. 12(c).Clearly, the proposed epipolar constraint model yields better results, and we conclude that it increases the accuracy of the feature-matching process.It should be pointed out that the proposed epipolar constraint model is used to constrain the target feature point to a line, which is different from a point-to-point matching method.

Conclusion
In many cases of using mirror virtual binocular vision systems, the same target is imaged using one camera and different reflections, leading to a mirror relation between the left and right parts of the captured image; in this situation, the two real cameras of the traditional binocular system are replaced by virtual cameras, which are formed by a single camera and mirrors.To perform the feature matching process effectively, a single-camera mirror feature-matching rule, i. e., a mirror epipolar constraint model, used in the 3-D reconstruction process, was established here for a mirror virtual binocular vision system.To validate the proposed epipolar constraint model and to evaluate its performance in practical applications, system calibration experiments, error analysis, and realistic feature-matching experiments were performed using a virtual binocular stereo vision system and the results of these experiments were analyzed and reported.The results showed that the proposed epipolar constraint method is feasible and can increase the accuracy of feature matching.

Fig. 2
Fig. 2 Mirror binocular stereo vision system: (a) imaging principle of real and virtual cameras and (b) real calibration plane image captured by the system.

Fig. 3
Fig.3Multimirror binocular stereo vision system: (a) imaging principle of two virtual cameras, (b) an image captured by the system, and (c) an amplified view.

Fig. 4
Fig. 4 Epipolar point's position in the mirror binocular vision system.

E
Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 1 ; 6 3 ; 4 3 2 sð pl − pr Þ ¼ aM 1 u sð pr − ẽÞ ¼ bM 1 u :

Fig. 5
Fig. 5 Single-camera epipolar constraint principle of the mirror binocular vision system.

Fig. 7 Fig. 8
Fig. 7 Single-camera epipolar constraint principle for one and two reflections.

Fig. 9
Fig. 9 Calibration images used for TSCM: (a) and (b) two of eight images for calibration of intrinsic parameters and (c) single image for calibration of structural parameters.

Fig. 10
Fig. 10 Four calibration images used for Zhou et al.'s method.

Fig. 11
Fig. 11 Image processing in validation experiment: (a) the tested source image, (b) extraction of feature point coordinates, (c) lines based on feature points and epipolar point, and (d) reconstruction of all feature points using the matching results of the proposed model.

Table 1
System calibration results using two methods.

Table 2
Measured distance of adjacent feature point.
• Vol.56(8) Chai, Zhou, and Chen: Epipolar constraint of single-camera mirror binocular stereo vision systems Downloaded From: https://www.spiedigitallibrary.org/journals/Optical-Engineering on 15 Sep 2023 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use results of the four experiments are listed in Table