Pose estimation method for planar mirror based on one-dimensional target

Wenlei Liu; Sentang Wu; Xiaolong Wu

doi:10.1117/1.OE.57.7.073101

5 July 2018 Pose estimation method for planar mirror based on one-dimensional target

Wenlei Liu, Sentang Wu, Xiaolong Wu

Author Affiliations +

Optical Engineering, Vol. 57, Issue 7, 073101 (July 2018). https://doi.org/10.1117/1.OE.57.7.073101

Abstract

This paper presents a pose estimation method for planar mirror based on a one-dimensional (1-D) target. First, a 1-D target is maneuvered in front of a camera so as to allow the camera to capture several images of both the original and virtual feature points on the target. Next, the normal vector of the mirror and the distance between the mirror and the camera are calculated by a closed-form method. Last, a nonlinear optimization is used to refine the closed-form solution by minimizing the reprojection error of reflection projection. Synthetic data and real experiments show that the proposed method is accurate and reliable. The proposed method is simple and flexible. No prior information about the target motion and camera pose is assumed. A minimum of two nonparallel target positions are required. It is suitable for the pose estimation of planar mirrors, especially when they are in fixed positions. We can find the posture of the fixed camera according to the normal vector of the plane mirror.

1. Introduction

Vision sensors have advantages of flexibility and noncontact measurement. Planar mirrors are utilized in a wide variety of applications, such as fringe-reflection measurement,¹ planar catadioptric vision sensors,² stereovision by a single camera, and calibration of multiple cameras with nonoverlapping fields of view (FOVs).³^,⁴ A common major challenge of these applications is the pose estimation of the planar mirror. It is important and necessary to develop accurate and flexible pose estimation methods for planar mirrors.

There has been some research on the pose estimation of planar mirrors. Lebraly et al.⁵ used a planar mirror to create an overlap between cameras by attaching markers on the mirror’s surface. However, pasting markers on the mirror was inconvenient and troublesome. Sturm et al.,⁶ Rodrigues et al.,⁷ Hesch et al.,⁸ and Kosuke Takahashi et al.⁹ proposed different pose estimation methods based on the fact that successive reflections in two-mirror planes were a fixed-axis rotation around the line of intersection of two mirror planes. Reference 10 proposed a practical visual inertial calibration method, which is based on visual observations taken in front of a planar mirror. Reference 11 presented a design and construction of a new system for stereovision with a single CCD camera. Reference 12 proposed a method for the calibration, which is used the orthogonal phase-shifted sinusoidal patterns during the calibration process.

Xiao et al.¹ calculated the transformation between the camera and the LCD screen by a markerless flat mirror. Li et al.¹³ measured the pose of a flat mirror by capturing the reflected structured light pattern. The translation vector was calculated by the mirror motion without any tilt. However, pure translations can hardly be achieved without external equipment. Ren et al.¹⁴ calibrated a stereodeflectometry system, where two cameras captured the fringe patterns produced by an LCD screen reflected by a flat mirror. The above methods require that the mirror is placed at several positions. However, these methods fail when the camera and the mirror are in fixed positions.

In Ref. 15, an algorithm is based on an orthogonality constraint, which should be satisfied by all families of reflections of a single reference object. It can return a unique solution of three P3P problems from three mirrored images. Reference 16 presents a method that utilizes reasonable assumptions about geometric features such as a plane or an intersecting line of two planes. It obtains an initial solution via singular value decomposition (SVD), which is then refined via nonlinear optimization, which does not require any assumption about structures but estimates the parameters of geometric features. In Ref. 17, an accurate closed-form solution with explicit geometric meanings is proposed. It minimizes an error metric based on all reflections of rotation, and then derives the optimal solution that gives the real camera rotation directly by SVD of the sum of mirrored camera rotations. Then, it can minimize object-space collinearity error.

This paper is also inspired by Shah¹⁸ and Li et al.¹⁷ Shah found the best rotation matrix with determinant 1. In contrast, we seek the best orthogonal matrix with determinant $- 1$ , and the best reflection matrix can be constructed by a matrix with determinant $- 1$ . Li et al.¹⁷ calculated the camera rotation by minimizing the error of all rotation reflections. In this paper, we initialize the normal vector by minimizing the error of reflections of the unit direction vectors.

In this paper, we focus on the pose estimation of the planar mirror based on a one-dimensional (1-D) target. 1-D target has advantages of simple structure and flexibility. First, instead of moving the mirror, we move the target in front of the camera to provide multiple (at least two) views of both the original and virtual targets. The camera and the mirror are fixed during the process. Next, the normal vector of the planar mirror is estimated based on the direction vectors of the original and virtual targets. The distance between the camera and the mirror is calculated based on reflection equations. Last, the closed-form solution thus obtained is refined using nonlinear optimization by minimizing the reprojection error of reflection projection. Compared with conventional methods, the proposed method is suitable for the pose estimation of planar mirrors, especially when camera and the mirror are in fixed positions. After obtaining the position information in the plane mirror, the camera can be observed the carrier virtual image through plane mirror, then the rotation matrix, and translation vector of the image machine coordinate system ${C}$ and the carrier coordinate system ${B}$ can be calculated. The calibration of the installation parameters of the image machine is completed.

The rest of this paper is organized as follows: preliminary work is introduced in Sec. 2. The principle of the proposed method is described in Sec. 3. Synthetic data and real-data experiments are performed in Sec. 4. The conclusions are given in Sec. 5.

2. Preliminaries

2.1.

Camera Model

Perspective projection is used as the camera model. The parameters and coordinates of the perspective projection model are shown in Fig. 1. The relation between a spatial point $P = {[X, Y, Z]}^{T}$ in ${C}$ and the corresponding image point $p = {[u, v]}^{T}$ can be described as follows:

Eq. (1)

s [\begin{matrix} p \\ 1 \end{matrix}] = K P, K = [\begin{matrix} f_{x} & 0 & u_{0} \\ 0 & f_{y} & v_{0} \\ 0 & 0 & 1 \end{matrix}],

where

s

is a scale factor,

K

is the intrinsic matrix in camera.

f_{x}

and

f_{y}

are the equivalent focal lengths in horizontal and vertical directions, respectively. (

u_{0}, v_{0}

) denotes the principal point.

Fig. 1

The perspective projection model and coordinate relation.

As shown in Fig. 2(a), both the original and the virtual targets are in the camera’s FOV. $O_{c} x_{c} y_{c} z_{c}$ is the coordinate frame of the camera, denoted as ${C}$ .

Fig. 2

The 1-D target is placed in front of the camera. (a) Projections of the original and virtual targets at the $i$ ’th position and (b) reflection through the planar mirror.

2.2.

Pose Estimation of One-Dimensional Target

As shown in Fig. 2, $P_{i, j}$ and ${\tilde{P}}_{i, j}$ denote the spatial coordinates of the $j$ ’th original and the $j$ ’th virtual feature points at the $i$ ’th position in ${C}$ , respectively. $p_{i, j}$ and ${\tilde{p}}_{i, j}$ are the corresponding image coordinates, $i = 1, \dots, M$ , $j = 1, \dots, N$ . $M$ is the number of target positions and $N$ is the number of feature points on the target.

The original target and the virtual target at $i$ ’th position are uniquely characterized by $P_{i, 1}$ , $d_{i}$ and ${\tilde{P}}_{i, 1}, {\tilde{d}}_{i}$ , respectively. $d_{i}$ and ${\tilde{d}}_{i}$ are the unit direction vectors of the original and virtual targets at $i$ th position, respectively.

For the $i$ th position, $P_{i, 1}$ and $d_{i}$ can be obtained by the method proposed by Liu et al.¹⁹^,²⁰ It requires that at least three collinear feature points on the target. Suppose $P_{i, 1} (x_{1}, y_{1}, z_{1})$ , $P_{i, 2} (x_{2}, y_{2}, z_{2})$ , and $P_{i, 3} (x_{3}, y_{3}, z_{3})$ , then

Eq. (2)

\frac{x_{3} - x_{1}}{x_{2} - x_{1}} = \frac{y_{3} - y_{1}}{y_{2} - y_{1}} = \frac{z_{3} - z_{1}}{z_{2} - z_{1}} = ρ,

where

ρ = ‖ P_{i 3} - P_{i 1} ‖ / ‖ P_{i 2} - P_{i 1} ‖

can be computed from the distances between the points

P_{i, 1}

,

P_{i, 2}

,

P_{i, 3}

.

Suppose that in the image coordinate system, $p_{i, 1} (u_{1}, v_{1})$ , $p_{i, 2} (u_{2}, v_{2})$ , and $p_{i, 3} (u_{3}, v_{3})$ . Combining Eqs. (1) and (2), we can obtain

Eq. (3)

{\begin{matrix} (u_{1} - u_{0}) z_{1} = x_{1} f_{x} \\ (v_{1} - v_{0}) z_{1} = y_{1} f_{y} \\ (u_{2} - u_{0}) z_{2} = x_{2} f_{x} \\ \begin{matrix} (v_{2} - v_{0}) z_{2} = y_{2} f_{y} \\ (u_{3} - u_{0}) z_{3} = x_{3} f_{x} \\ (v_{3} - v_{0}) z_{3} = y_{3} f_{y} \\ \begin{array}{l} (ρ - 1) x_{1} = ρ x_{2} - x_{3} \\ (ρ - 1) y_{1} = ρ y_{2} - y_{3} \\ (ρ - 1) z_{1} = ρ z_{2} - z_{3} \end{array} \end{matrix} \end{matrix} .

By solving Eq. (3), the coordinates of $p_{i, 1} (x_{1}, y_{1}, z_{1})$ , $p_{i, 2} (x_{2}, y_{2}, z_{2})$ , and $p_{i, 3} (x_{3}, y_{3}, z_{3})$ can be retrieved and the initial value of the normal vector $d_{i}$ can be obtained. Due to the image noise, if there are more than three feature points, results of different point triplets, e.g., ( $P_{i, 1}, P_{i, 2}, P_{i, 3}$ ) and ( $P_{i, 1}, P_{i, 2}, P_{i, 4}$ ), are not equivalent.

Let ${\hat{p}}_{i, j}$ be the image coordinates of the reprojection feature point

Eq. (4)

s [\begin{matrix} {\hat{p}}_{i, j} \\ 1 \end{matrix}] = K [P_{i, 1} + (j - 1) L d_{i}],

where

s

is a scale factor,

L

is the known interval of two adjacent feature points on the target, and

K

denotes the intrinsic matrix in camera.

To utilize the redundant information of feature points, $P_{i, 1}$ and $d_{i}$ are refined by minimizing the following reprojection error using Levenberg–Marquardt algorithm:²¹

Eq. (5)

f (Ω) = \sum_{j = 1}^{N} d^{2} (p_{i, j}, {\hat{p}}_{i, j}),

where

Ω = (P_{i, 1}, d_{i})

.

d_{i} = {[\cos θ_{i} \cos φ_{i}, \sin φ_{i}, \cos θ_{i} \sin φ_{i}]}^{T}

.

θ_{i}

and

φ_{i}

are the pitch angle and yaw angle of the

i

’th unit direction vector with respect to

{C}

, respectively.

d (\cdot)

is the distance function between two points. The initial value of (

P_{i, 1}, d_{i}

) can be obtained by Liu’s method.¹⁹

The pose of the $i$ ’th virtual target $({\tilde{P}}_{i, 1}, {\tilde{d}}_{i})$ can also be obtained similarly.

2.3.

Planar Mirror Reflection

As shown in Fig. 2(b), camera ${C}$ is in front of a planar mirror $Π$ . The mirror is uniquely determined by its unit normal vector $n$ and the distance $l$ with respect to ${C}$ .²^,¹⁷ A spatial point $P$ lies on the mirror plane if and only if it satisfies

Eq. (6)

n^{T} P = l .

The Householder matrix $H$ ²²^,²³ is widely used in the mirror reflection model, and its equation is as follows:

Eq. (7)

H = I - 2 {nn}^{T} .

The Householder transformation is a linear transformation that transforms a vector into a mirror image reflected by a hyperplane. The Householder transformation can zero some elements of a vector and keep the norm of the vector constant.

The schematic diagram of a mirror surface emission model is shown in Fig. 3. The equation of the plane $Π$ is as follows:

Eq. (8)

n^{T} x = 0 \forall x \in Π .

Fig. 3

The schematic diagram of a mirror surface emission model.

If $x \in Π$ , then

Eq. (9)

H x = (I - 2 {nn}^{T}) x = x .

If $y \notin Π$ , then

Eq. (10)

H y = (I - 2 {nn}^{T}) (x + k n) = x + k n - 2 k {nn}^{T} n = x - k n = \tilde{y} .

The reflection in $Π$ can be represented by the following matrix:

Eq. (11)

S = [\begin{matrix} H & 2 \ln \\ 0 & 1 \end{matrix}] .

Through the conversion of $S$ can obtain the position of the virtual mirror $\tilde{P}$ corresponding to the space point $P$ in the camera coordinate system. The space point $P$ and the corresponding virtual point are related as follows:

Eq. (12)

[\begin{matrix} \tilde{P} \\ 1 \end{matrix}] = S [\begin{matrix} P \\ 1 \end{matrix}] .

3. Principle of the Mirror Pose Estimation

3.1.

Analysis of the Unique Solution

The calculation of the normal vector $n$ is equivalent to solving the Householder matrix $H$ . According to Eqs. (11) and (12), $d_{i}$ and ${\tilde{d}}_{i}$ are related by the Householder matrix as follows:

Eq. (13)

{\tilde{d}}_{i} = {Hd}_{i} .

From Eq. (13), one position is not enough to obtain a unique solution. For two positions, we have

Eq. (14)

{\tilde{d}}_{1} = {Hd}_{1}, {\tilde{d}}_{2} = {Hd}_{2} .

According to the principle of mirror image, assume that target real image is positioned for the right-handed coordinate system, and in the target virtual image, the left and right, upside down, and up and down directions remain the same with respect to the left-hand target image coordinate system.⁴ Therefore, the relationship between real image $d_{1} {\times d}_{2}$ and virtual image ${\tilde{d}}_{1} {\times \tilde{d}}_{2}$ is as follows:

Eq. (15)

- {\tilde{d}}_{1} \times {\tilde{d}}_{2} = H (d_{1} \times d_{2}) .

Combining Eqs. (14) and (15), we have

Eq. (16)

[\begin{matrix} {\tilde{d}}_{1} & {\tilde{d}}_{2} & - \tilde{d} \times {\tilde{d}}_{2} \end{matrix}] = H [\begin{matrix} d_{1} & d_{2} & d_{1} \times d_{2} \end{matrix}] .

If $d_{1}$ is not parallel to $d_{2}$ , matrix $[\begin{matrix} d_{1} & d_{2} & d_{1} \times d_{2} \end{matrix}]$ is invertible. Thus, $H$ can be uniquely determined as follows:

Eq. (17)

H = [\begin{matrix} {\tilde{d}}_{1} & {\tilde{d}}_{2} & - \tilde{d} \times {\tilde{d}}_{2} \end{matrix}] {[\begin{matrix} d_{1} & d_{2} & d_{1} \times d_{2} \end{matrix}]}^{- 1} .

If $d_{1}$ is parallel to $d_{2}$ , the target motion is a pure translation, det $[\begin{matrix} d_{1} & d_{2} & d_{1} \times d_{2} \end{matrix}] = 0$ , thus a unique solution cannot be found.

3.2.

Computing the Normal Vector n

If the target is placed more than two times, the best Householder matrix can be computed from the solution to the orthogonal Procrustes problem.²¹ According to Eq. (13), the sum of the normal vector error is

Eq. (18)

E = \sum_{i = 1}^{M} e_{i} = \sum_{i = 1}^{M} {‖ {\tilde{d}}_{i} - H d_{i} ‖}_{F}^{2},

where

{‖ \cdot ‖}_{F}

denotes the Frobenius norm,

{‖ x ‖}_{F}^{2} = tr (x^{T} x) = tr (x x^{T})

. If measurements are noiseless,

E = 0

.

The best approximation of matrix $H$ is equivalent to the matrix that solves the minimization problem of Eq. (18). According to the properties of the Householder matrix, Eq. (18) is expanded as

Eq. (19)

E = \sum_{i = 1}^{M} tr [({\tilde{d}}_{i}^{T} - d_{i}^{T} H^{T}) ({\tilde{d}}_{i} - H d_{i})] = 2 M - 2 tr (H Δ),

where

Δ = \sum_{i = 1}^{M} d_{i} {\tilde{d}}_{i}^{T}

.

It can be seen that minimization of $E$ is equivalent to maximize $tr (H Δ)$ . Therefore, the matrix $H$ that solves the minimization problem of Eq. (18) is equivalent to $H$ that solves

Eq. (20)

\max_{H} tr (H Δ) .

There are some researches¹⁸^,²² on finding the best rotation matrix with determinant $+ 1$ . However, the Householder matrix is not a rotation but actually a reflection with determinant $- 1$ . To construct the best reflection, a lemma is introduced.

Lemma 1

For a given $3 \times 3$ matrix $Δ$ and a Householder matrix $H$

Eq. (21)

tr (H Δ) \leq tr (D Σ),

where

U Σ V^{T}

is the full SVD of

Δ

, the singular values are in decreasing order, and

Eq. (22)

D = diag [1, 1, - det (V U^{T})] .

Proof

Using $tr (A B) = tr (B A)$ and the SVD result of matrix $Δ$ , we have

Eq. (23)

tr (H Δ) = tr (H U Σ V^{T}) = (V^{T} H U D D Σ),

where

D^{2} = I

is the identity matrix. Let

\hat{H} = V^{T} H U D

.

Eq. (24)

\hat{H} {\hat{H}}^{T} = (V^{T} H U D) {(V^{T} H U D)}^{T} = I,

Eq. (25)

\det (\hat{H}) = \det (V^{T} H U D) = \det (V^{T}) \det (H) \det (U) \det (D) = - [\det {(V^{T})}^{2}] {[\det (U)]}^{2} \det (H) = 1 .

According to Eqs. (24) and (25), we can come to the conclusion that $\hat{H}$ is an orthogonal matrix with determinant 1 and hence a rotation matrix.

Suppose that $\hat{H} = [\begin{matrix} h_{11} & \dots & h_{1 j} \\ ⋮ & ⋱ & ⋮ \\ h_{i 1} & \dots & h_{i j} \end{matrix}]$ , $Σ = diag (σ_{1}, σ_{2}, σ_{3})$ , we have

Eq. (26)

tr (H Δ) = tr (\hat{H} D Σ) = σ_{1} h_{11} + σ_{2} h_{22} - σ_{3} h_{33} \det (V U^{T}) .

Therefore, the trace is maximized if $h_{i i} = 1$ . As $\hat{H}$ is an orthogonal matrix, this means that $\hat{H}$ would have to be the identity matrix. Then

Eq. (27)

tr (H Δ) \leq σ_{1} + σ_{2} - σ_{3} det (V U^{T}) = tr (D Σ) .

According to two sets of the original and virtual direction vectors, the best reflection matrix can be obtained by Theorem 1.

Theorem 1.

The solution to the maximization problem of Eq. (20) is as follows:

Eq. (28)

H = V D U^{T},

where

U Σ V^{T}

is the full SVD of

Δ

with decreasing singular values and

D

is subject to Eq. (22).

Proof.

From Lemma 1, the maximization problem of Eq. (20) is achieved if a Householder matrix $H$ can be constructed such that

Eq. (29)

tr (H Δ) = tr (D Σ) .

Let

H = V D U^{T}

, then

tr (H Δ) = tr (V D U^{T} U Σ V^{T}) = tr (D Σ) .

When the Householder matrix $H$ is determined, the unit normal vector $n$ is obtained.

3.3.

Computing the Distance l

From Eq. (12), we have

Eq. (30)

2 \ln = {\tilde{P}}_{i, j} - {HP}_{i, j} .

Therefore, the distance $l$ can be computed as

Eq. (31)

l = \frac{1}{2 M N} \sum_{i = 1}^{M} \sum_{j = 1}^{N} [n^{T} ({\tilde{P}}_{i, j} - {HP}_{i, j})],

where

P_{i, j}

and

{\tilde{P}}_{i, j}

can be calculated according to the Eq. (3).

H

can be calculated according to Eq. (17).

3.4.

Nonlinear Optimization

The results obtained in Sec. 3.2 and Sec. 3.3, which were called as the closed-form solution. The closed-form calculation is fast, but it is sensitive to the noise. In this subsection, the closed-form solution is refined with nonlinear optimization by minimizing the reprojection error of reflection. According to the camera model, we have

Eq. (32)

s [\begin{matrix} {\overset{⌢}{p}}_{i, j} \\ 1 \end{matrix}] = K [\begin{matrix} I & 0 \end{matrix}] [\begin{matrix} I - 2 {nn}^{T} & 2 \ln \\ 0 & 1 \end{matrix}] [\begin{matrix} p_{i, j} \\ 1 \end{matrix}],

where s is a scale factor,

K

denotes the intrinsic matrix in camera,

I

is a

3 \times 3

identity matrix, and 0 is a

1 \times 3

zero vector.

{\overset{⌢}{p}}_{i, j}

is the reprojected image coordinates.

The mirror normal vector $n$ and the distance $l$ are refined by minimizing the following function using the Levenberg–Marquardt algorithm²¹

Eq. (33)

f (Ω) = \sum_{i = 1}^{M} \sum_{j = 1}^{N} ω^{2} ({\tilde{p}}_{i, j}, {\overset{͡}{p}}_{i, j}),

where

Ω = (n, l)

,

n = {[\cos θ \cos φ, \sin θ, \cos θ \sin φ]}^{T}

.

θ

and

φ

are the pitch angle and yaw angle of the normal vector

n

with respect to

{C}

, respectively.

ω (\cdot)

is the distance between image points and

{\tilde{p}}_{i j}

is the image coordinate corresponding to the imaginary point

{\overset{⌢}{p}}_{i, j}

. After optimization, the optimal value of the normal vector

n

and the distance

l

is obtained.

3.5.

Calculating the Relative Position of the Image Machine ${C}$ and the Carrier ${B}$

Figure 4 shows the relationship diagram of the vector coordinate system ${B}$ , the vector image frame ${B^{'}}$ , and the machine coordinate system ${C}$ , where the $R_{BC}$ and $R_{B^{'} C}$ stand for the rotation vector, the $t_{BC}$ and $t_{B^{'} C}$ represent the translation vector, and the $S$ mean mirror reflection matrix.

Fig.4

The relationship diagram of the vector coordinate system ${B}$ , the vector image frame ${B^{'}}$ , and the machine coordinate system ${C}$ .

The image of the mounting carrier in the mirror is observed by the plane mirror, so as to obtain the transformation relationship between the plane frame ${C}$ and the vector virtual frame ${B^{'}}$ . According to the above results, the transformation relationship between the image coordinate system ${C}$ and the carrier image coordinates ${B}$ can be obtained directly. Thus, the calibration of the installation parameters of the imager can be completed.

Eq. (34)

[\begin{matrix} R_{B C} & t_{B C} \\ 0 & 1 \end{matrix}] = [\begin{matrix} R_{B^{'} C} & t_{B^{'} C} \\ 0 & 1 \end{matrix}] [\begin{matrix} I - 2 n n^{T} & 2 \ln \\ 0 & 1 \end{matrix}],

where

R

is the rotation matrix,

t

is the translation vector, subscripts

B^{'} C

and

B C

, respectively, represent the conversion relationship between the camera coordinate system

{C}

and the virtual carrier coordinate

{B^{'}}

, and the transformation relationship between the camera coordinate system

{C}

and the carrier image coordinates

{B}

.

3.6.

Method Summary

The calibration method of the installation parameters of the image machine based on the plane mirror is proposed in this section, which keeps the relative position of the image machine and the plane mirror unchanged. The main processes of the proposed method are as follows:

Step 1: The intrinsic parameters of the camera are calibrated using the Bouguet Camera Calibration Toolbox based on Zhang’s method.²⁴^,²⁵ The internal parameters of the camera mainly include the following: $f_{x}, f_{y}, u_{0}, v_{0}$ .
Step 2: The 1-D target with $N$ feature points is placed in front of the camera for $M$ times. For each position, one image is taken by the camera. Both the original and virtual targets are in the camera’s FOV. All the images are undistorted according to the intrinsic calibration results. Thus, image coordinates are distortionless.
Step 3: The optimal estimation value of matrix $H$ can be obtained by Orthogonal Procrustes problem. For the $i$ th position, compute the value of $P_{i, 1}, d_{i}$ and ${\tilde{P}}_{i, 1}, {\tilde{d}}_{i}$ , respectively. Repeat this step for $i = 1, \dots, M$ .
Step 4: By obtaining the minimum value of the reprojection error function, the optimal value of the mirror normal vector $n$ and the image machine-mirror distance $l$ are obtained.
Step 5: The rotation matrix $R_{BC}$ and the translation vector $t_{BC}$ between the image machine coordinate system and the carrier coordinate system are calculated, and the calibration of the installation parameters of the image machine is completed.

4. Experimental Results

Through the previous derivation, the mirror position (the mirror vector $n$ and the image machine-mirror distance $l$ ) are the key parameters in the calibration process of the installation parameters of the image machine. Therefore, this section takes the pose error of plane mirror as the accuracy evaluation index, analyzes the related factors that affect the accuracy through simulation data and physical experiments, and compares it with other calibration methods.

4.1.

Simulation Data

To evaluate the accuracy, the estimation errors are defined as follows:

(a) Error of the mirror normal vector $n$ against the truth value $n_{g}$ is defined as follows:
Eq. (35)
$E_{n} = arc \cos (n^{T} n_{g}) .$
(b) Error of the distance $l$ against the truth value $l_{g}$ is defined as follows:
Eq. (36)
$E_{l} = l - l_{g} .$
(c) Reprojection error is defined as follows:
Eq. (37)
$E_{q} = \sqrt{\sum_{i = 1}^{M} \sum_{j = 1}^{N} \frac{{‖ {\tilde{p}}_{i, j} - {\hat{p}}_{i, j} ‖}^{2}}{2 M N}} .$

In this experiment, one simulated camera and a planar mirror are used to construct the system. The intrinsic parameters of the camera are $f_{x} = f_{y} = 995.556$ , $u_{0} = 512$ , $v_{0} = 384$ . The image resolution is $1024 pixel \times 768 pixel$ . The mirror normal vector $n = {[- 0.5, 0, 0.866]}^{T}$ , i.e., $θ = 0$ , $φ = 120 \deg$ . The distance $l = 400 mm$ , the interval $L = 25 mm$ .

The positions of target are randomly generated. Only the original and virtual targets in the camera’s FOV are used. Gaussian noise with zero mean and different standard deviation σ is added to the image coordinates of feature points. The effects of several factors on the accuracy of the pose estimation are evaluated quantitatively by synthetic data as follows.

4.1.1.

Influence of the characteristic point noise on the accuracy

In this experiment, $M = 10$ , $N = 9$ . Gaussian noise is added to the image coordinates of the feature points. The noise level varies from 0 to 1.0 pixel. For each level, 100 independent trials are performed to compute the relative errors of $E_{n}$ , $E_{l}$ and RMS errors of $E_{q}$ .

From Fig. 5, the error increases linearly with the noise level. It indicates that the closed-form solution can be effectively adjusted by the proposed method. If the noise level of a system is $< 0.5 pixel$ , relative errors of the normal vector $n$ and the distance $l$ are $< 0.06 %$ and 0.5%, respectively, which are applicable for common applications.

Fig. 5

Error of pose estimation versus the noise level: (a) the relative error of En versus noise level, (b) the relative error of El versus noise level, and (c) the RMS error of Eq versus noise level.

4.1.2.

Influence of the number of target placement on the accuracy

In this experiment, Gaussian noise with $σ = 0.2 pixel$ is added to the image coordinates of the feature points. $M$ varies from 2 to 12, $N$ is set to 9. For each level, 100 independent trials are performed and are used to compute relative errors.

From Fig. 6, the error decreases with the increasing number of target positions. When the target is placed only twice, the pose estimation error is rather high. The estimation error decreases rapidly as the number of positions increases. When there are enough positions, the contribution of additional positions decreases.

Fig. 6

Error of pose estimation vs. the number of target positions M (a) the relative error of En versus the number of target positions, (b) the relative error of El versus the number of target positions, and (c) the RMS error of Eq versus the number of target positions.

4.1.3.

Influence of the number of target feature points on the accuracy

In this experiment, Gaussian noise with $σ = 0.2 pixel$ is added to the image coordinates of the feature points. $M$ is set to 10, $N$ varies from 3 to 16. For each level, 100 independent trials are performed and are used to evaluate relative errors.

From Fig. 7, increasing number of feature points benefits the estimation accuracy. When there are only three feature points on the target, the estimation error reaches maximum. The accuracy improves sharply when $N$ increases from 3 to 8. When $N > 10$ , the accuracy stables at the certain level.

Fig. 7

Error of pose estimation versus the number of feature points N (a) the relative error of En versus the number of feature points, (b) the relative error of El versus the number of feature points, and (c) the RMS error of Eq versus the number of feature points.

4.2.

Physical Experiments

As shown in Fig. 8(a), a Canon 60D digital camera and a planar mirror are in fixed positions. The image resolution of the camera is $1920 pixel \times 1280 pixel$ . The intrinsic parameters of the vision sensor are calibrated using the Bouguet’s calibration toolbox,¹⁵ as shown in Table 1. We use the intersection of two rows of checkerboards equivalently to the 1-D target. The distance between neighboring feature points in the checkerboard is 20 mm, i.e., $L = 20 mm$ . The two-row checkerboard is used to define 1-D target for feature extraction.

Fig. 8

(a) An image captured by the camera. Both the original and virtual 1-D target are in the camera’s FOV and (b) a checkerboard is used to evaluate the accuracy.

Table 1

Intrinsic parameters of the camera.

	fx	fy	u0	v0
Camera	1548	1548	960	640
Bouguet	1561.293	1560.153	972.497	597.061
Error (%)	0.86	0.79	1.3	6.7

As shown in Fig. 8(b), a checkerboard is placed between the camera and the mirror. Both the original and virtual checkerboards are in the camera’s FOV. The spatial coordinates of the corner points of the checkerboards can be obtained by the Bouguet’s toolbox. In this experiment, reflective error $E_{s}$ and the reprojection error $E_{q}$ are used to evaluate the accuracy

Eq. (38)

E_{s} = \frac{1}{(n_{x} + 1) (n_{y} + 1)} \sum_{i = 1}^{n_{x + 1}} \sum_{j = 1}^{n_{y} + 1} ‖ {\tilde{Q}}_{i, j} - ({HQ}_{i, j} + 2 \ln) ‖,

where

Q_{i, j}

and

{\tilde{Q}}_{i, j}

are the spatial coordinates of the original and virtual corner points of the checkerboards, with respect to

{C}

.

n_{x}

and

n_{y}

are the number of squares along two orthogonal directions, respectively.

To verify the effect of several factors shown in the synthetic experiment, pose estimations of different position numbers and different feature points are implemented.

4.2.1.

Influence of the number of target placement on the accuracy

In this experiment, the number of target positions $M$ is changed from 2 to 12, the number of feature points $N$ is set to 9. From Fig. 9, the result is similar to the result of the synthetic experiment in Sec. 4.1.2. An increasing number of target positions benefits the accuracy, especially when the number of target positions is relatively small.

Fig. 9

Error of pose estimation versus the number of target positions M (a) the reflective error Es versus target position number and (b) the reprojection error Eq versus target position number.

4.2.2.

Influence of the number of target feature points on the accuracy

In this experiment, the number of target positions $M$ is set to 10, the number of feature points $N$ is changed from 3 to 12. From Fig. 10, it also indicates a similar result of the synthetic experiment in Sec. 4.1.3. The estimation error peaks at $N = 3$ , which corresponds to the minimum quantity of feature points. Then, the estimation error decreases sharply with the increasing number of feature points. When $N > 8$ , the estimation error flattens out gradually.

Fig. 10

Error of pose estimation versus the number of feature points N (a) the reflective error ES versus the number of feature points and (b) the reprojection error Eq versus the number of feature points.

4.2.3.

Discussions

Evidently, there exists a trade-off between the accuracy and the workload. According to the effect, analysis of the number of target positions $M$ and the number of feature points $N$ , $M = 10$ and $N = 9$ is a balanced choice. When the error of an image point is 0.5 pixel, the synthetic experiment shows that relative errors of the normal vector $n$ and the distance $l$ are 0.06% and 0.45%, respectively. It is acceptable for common applications such as the camera-to-base calibration.⁸

In addition, there is a gap between the results of synthetic data and real-data experiments. There may be some reasons for this. First, the feature point extraction method is a commonly used method with good generality and acceptable accuracy. Extraction algorithms with higher accuracy improve the accuracy, which will be further investigated. Second, the mirror used in this paper is a domestic appliance, not a precision optical device. Precision mirrors can be used to improve the accuracy.

4.3.

Comparisons

There are many methods for the pose estimation. Here, three typical methods are described as follows:

Method 1: This method is similar to the proposed method. As shown in Fig. 8(b), a checkerboard is used as the calibration target, instead of the 1-D target. The corner points are extracted and calibrated by the Bouguet’s toolbox. The pose of the mirror is also refined using nonlinear optimization. We use method 1 as the criterion for evaluating the merits of the proposed method.
Method 2: As shown in Fig. 11, a reference object is placed on the ground and remains fixed. Then, the pose of the mirror can be calculated by Li’s method.¹⁷

Fig. 11

A virtual image of the reference object.

For each method, 10 images are captured to estimate the mirror pose. The reflective error Es is used to evaluate the accuracy. The results of different methods are shown in Table 2.

Table 2

The estimation errors of different methods.

	Proposed method (M=10, N=9)	Closed-form solution (M=10, N=9)	Method 1	Method 2
$E_{s} / mm$	1.41	6.38	1.37	5.71

From Table 2, the comparisons show that the proposed method achieves a good accuracy. Compared with the closed-form solution, the closed solution is simple and fast, but it is more sensitive to noise and larger in error. The same line determined by the selection of different points is different. Therefore, the normal vectors determined by these lines will also have certain differences, which will produce certain errors. Compared with the checkerboard of method 1, the errors of the two methods are relatively small. However, the number of feature points of 1-D target is reduced a lot, under the condition that the computation accuracy is not decreased. Therefore, according to the 1-D target of the known size, the calibration of the imaging machine can be achieved without reducing the precision, and the limitation of the complex calibration tool can be removed, so that the calibration is more flexible and convenient. The accuracy of method 3 is roughly the same as the closed-form solution. However, method 3 fails when the camera and the mirror cannot be moved.

5. Conclusion

In this paper, a pose estimation method for planar mirrors is presented. The method only needs a simple 1-D target and a digital camera. First, the camera captures the target in different positions. The pose of the target in each position is obtained. Then, the normal vector and the distance between the camera and the mirror are calculated by a closed-form method, followed by a nonlinear optimization.

Our method is simple and flexible. A minimum of two nonparallel target positions and three collinear feature points on the target are needed. No prior information about the target motion and the camera’s pose is required. The proposed method is accurate and robust to noises. Both synthetic data and real experiments demonstrate the effectiveness of our method. In addition, both the camera and the mirror do not need to move during the process, so it is suitable for application where the camera and the mirror need to remain fixed.

In some applications, the transformation between the body frame and the camera frame need to be accurately calibrated. However, cameras mounted on the body usually cannot observe the body directly. We plan to apply our method for the camera-to-body calibration in the future.

Acknowledgments

The authors acknowledge the support by the Industrial Technology Development Program under Grant No. B1120131046.

References

1.

Y. Xiao, X. Su and W. Chen, “Flexible geometrical calibration for fringe-reflection 3D measurement,” Opt. Lett., 37 (4), 620 –622 (2012). https://doi.org/10.1364/OL.37.000620 OPLEDP 0146-9592 Google Scholar

2.

G. L. Mariottini et al., “Planar mirrors for image-based robot localization and 3-D reconstruction,” Mechatronics, 22 398 –409 (2012). https://doi.org/10.1016/j.mechatronics.2011.09.004 MECHER 0957-4158 Google Scholar

3.

X. Feng and D. Pan, “Research on the application of single camera stereo vision sensor in three-dimensional point measurement,” J. Mod. Opt., 62 (15), 1204 –1210 (2015). https://doi.org/10.1080/09500340.2015.1024775 JMOPEW 0950-0340 Google Scholar

4.

R. K. Kumar et al., “Simple calibration of non-overlapping cameras with a mirror,” in Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, 2602 –2608 (2008). Google Scholar

5.

P. Lebraly et al., “Flexible extrinsic calibration of non-overlapping cameras using a planar mirror: Application to vision-based robotics,” in Proc. of IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, (IROS, 2010), 5640 –5647 (2010). Google Scholar

6.

P. Sturm and T. Bonfort, “How to compute the pose of an object without a direct view?,” Lect. Notes Comput. Sci., 3852 21 –31 (2006). https://doi.org/10.1007/11612704_3 Google Scholar

7.

R. Rodrigues, J. P. Barreto and U. Nunes, “Camera pose estimation using images of planar mirror reflections,” Lect. Notes Comput. Sci., 6314 382 –395 (2010). https://doi.org/10.1007/978-3-642-15561-1_28 Google Scholar

8.

J. A. Hesch, A. I. Mourikis, S. I. Roumeliotis, “Mirror-based extrinsic camera calibration,” Algorithmic Foundations of Robotics VIII, 57 Springer, Berlin, Heidelberg (2009). https://doi.org/10.1007/978-3-642-00312-7_18 Google Scholar

9.

K. Takahashi, S. Nobuhara and T. Matsuyama, “Mirror-based camera pose estimation using an orthogonality constraint,” IPSJ Trans. Comput. Vision Appl., 8 11 –19 (2016). https://doi.org/10.2197/ipsjtcva.8.11 Google Scholar

10.

G. Panahandeh, M. Jansson and P. Händel, “Calibration of an IMU-camera cluster using planar mirror reflection and its observability analysis,” IEEE Trans. Instrum. Meas., 64 (1), 75 –88 (2015). https://doi.org/10.1109/TIM.2014.2329388 Google Scholar

11.

T. P. Pachidis and J. N. Lygouras, “Pseudostereo-vision system: a monocular stereo-vision system as a sensor for real-time robot applications,” IEEE Trans. Instrum. Meas., 56 (6), 2547 –2560 (2007). https://doi.org/10.1109/TIM.2007.908231 Google Scholar

12.

Y. Xu et al., “A calibration method for non-overlapping cameras based on mirrored absolute phase target,” Int. J. Adv. Manuf. Technol., (2018). https://doi.org/10.1007/s00170-018-1704-8 Google Scholar

13.

L. Li et al., “Flat mirror tilt and piston measurement based on structured light reflection,” Opt. Express, 22 (22), 27707 –27716 (2014). https://doi.org/10.1364/OE.22.027707 OPEXFF 1094-4087 Google Scholar

14.

H. Ren, F. Gao and X. Jiang, “Iterative optimization calibration method for stereo deflectometry,” Opt. Express, 23 (17), 22060 –22068 (2015). https://doi.org/10.1364/OE.23.022060 OPEXFF 1094-4087 Google Scholar

15.

J. Y. Bouguet, “Camera calibration toolbox for Matlab,” (2013) http://www.vision.caltech.edu/bouguetj Google Scholar

16.

T. Matsuyama, S. Nobuhara and K. Takahashi, “A new mirror-based extrinsic camera calibration using an orthogonality constraint,” in IEEE Conf. on Computer Vision and Pattern Recognition, 1051 –1058 (2012). Google Scholar

17.

X. Li et al., “Accurate mirror-based camera pose estimation with explicit geometric meanings,” Sci. China-Technol. Sci., 57 (12), 2504 –2513 (2014). https://doi.org/10.1007/s11431-014-5713-z Google Scholar

18.

M. Shah, “Comparing two sets of corresponding six degree of freedom data,” Comput. Vis. Image Understanding, 115 (10), 1355 –1362 (2011). https://doi.org/10.1016/j.cviu.2011.05.007 CVIUF4 1077-3142 Google Scholar

19.

Z. Liu et al., “Novel calibration method for non-overlapping multiple vision sensors based on 1D target,” Opt. Lasers Eng., 49 (4), 570 –577 (2011). https://doi.org/10.1016/j.optlaseng.2010.11.002 Google Scholar

20.

Z. Liu, F. J. Li and G. J. Zhang, “An external parameter calibration method for multiple cameras based on laser rangefinder,” Measurement, 47 954 –962 (2014). https://doi.org/10.1016/j.measurement.2013.10.029 0263-2241 Google Scholar

21.

J. J. Moré, “The Levenberg–Marquardt algorithm: implementation and theory,” Numerical Analysis, 105 –116 Springer(1978). Google Scholar

22.

F. Zhang, “Matrix analysis and applications,” Int. J. Inf. Syst. Sci., 309 (1), i (2004). Google Scholar

23.

K. Kanatani, “Analysis of 3-D rotation fitting,” IEEE Trans. Pattern Anal. Mach. Intell., 16 (5), 543 –549 (1994). https://doi.org/10.1109/34.291441 ITPIDJ 0162-8828 Google Scholar

24.

Z. Y. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell., 22 (11), 1330 –1334 (2000). https://doi.org/10.1109/34.888718 ITPIDJ 0162-8828 Google Scholar

25.

Y. Bok et al., “Extrinsic calibration of non-overlapping camera-laser system using structured environment,” in IEEE/RSJ Int. Conf. on Intelligent Robots & Systems, 436 –443 (2014). Google Scholar

Biography

Wenlei Liu received his BS degree at Shandong University of Science and Technology, Qingdao, in 2012 and his MS degree at Beihang University, Beijing, in 2016. Currently, he is pursuing his PhD at Beihang University, Beijing, China. He mainly studies the high-precision visual relative navigation, and also studies the data fusion of multi data sensors to achieve the purpose of cooperative detection and guidance.

Sentang Wu received his PhD in dynamics, ballistics, and aircraft motion control systems from National Aviation University, Ukraine, in 1992. Currently, he is a professor of automation science and electrical engineering, and a PhD tutor at Beihang University, Beijing, China. He is also the navy missile expert in the National Defense Basic Research Institute and a member of the academic committee. His research interests include the theory and application of nonlinear stochastic systems, computer information processing and control, aircraft cooperative control, precision, and guidance.

Xiaolong Wu received his BS degree in Sichuan University, Chengdu, in 2007 and his PhD in Beihang University, Beijing, in 2017. He mainly studies the information acquisition system of the missile’s autonomous formation, including the relative navigation method of the missile's autonomous formation, the calibration of the vision sensor’s installation parameters and the optimization of the field of view, and the cooperative detection and guidance of the missile's autonomous formation.

CC BY: © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation Download Citation

Wenlei Liu, Sentang Wu, and Xiaolong Wu "Pose estimation method for planar mirror based on one-dimensional target," Optical Engineering 57(7), 073101 (5 July 2018). https://doi.org/10.1117/1.OE.57.7.073101

Received: 28 March 2018; Accepted: 12 June 2018; Published: 5 July 2018

Access the abstract

JOURNAL ARTICLE
10 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 2 scholarly publications.

Explore citations on Lens.org

KEYWORDS

Mirrors

Cameras

Calibration

Error analysis

Imaging systems

Optical engineering

Fiber Bragg gratings

1.

Introduction

2.

Preliminaries

2.1.

Camera Model

Eq. (1)

Fig. 1

Fig. 2

2.2.

Pose Estimation of One-Dimensional Target

Eq. (2)

Eq. (3)

Eq. (4)

Eq. (5)

2.3.

Planar Mirror Reflection

Eq. (6)

Eq. (7)

Eq. (8)

Fig. 3

Eq. (9)

Eq. (10)

Eq. (11)

Eq. (12)

3.

Principle of the Mirror Pose Estimation

3.1.

Analysis of the Unique Solution

Eq. (13)

Eq. (14)

Eq. (15)

Eq. (16)

Eq. (17)

3.2.

Computing the Normal Vector n

Eq. (18)

Eq. (19)

Eq. (20)

Lemma 1

Eq. (21)

Eq. (22)

Proof

Eq. (23)

Eq. (24)

Eq. (25)

Eq. (26)

Eq. (27)

Theorem 1.

Eq. (28)

Proof.

Eq. (29)

3.3.

Computing the Distance l

Eq. (30)

Eq. (31)

3.4.

Nonlinear Optimization

Eq. (32)

Eq. (33)

3.5.

Calculating the Relative Position of the Image Machine {C} and the Carrier {B}

Fig.4

Eq. (34)

3.6.

Method Summary

4.

Experimental Results

4.1.

Simulation Data

Eq. (35)

Eq. (36)

Eq. (37)

4.1.1.

Influence of the characteristic point noise on the accuracy

Fig. 5

4.1.2.

Influence of the number of target placement on the accuracy

Fig. 6

4.1.3.

Calculating the Relative Position of the Image Machine ${C}$ and the Carrier ${B}$