Flexible decoupled camera and projector fringe projection system using inertial sensors

Abstract. Measurement of objects with complex geometry and many self-occlusions is increasingly important in many fields, including additive manufacturing. In a fringe projection system, the camera and the projector cannot move independently with respect to each other, which limits the ability of the system to overcome object self-occlusions. We demonstrate a fringe projection setup where the camera can move independently with respect to the projector, thus minimizing the effects of self-occlusion. The angular motion of the camera is tracked and recalibrated using an on-board inertial angular sensor, which can additionally perform automated point cloud registration.


Introduction
Structured light systems (SLS) 1,2 are used to perform dimensional measurements on objects by projecting patterns of light, which are subsequently recorded by a camera. SLS have recently proliferated in industrial metrology, because they do not contact the object being measured and thus can be used with final-stage products; they operate much faster than standard contact coordinate measuring machines 3 and provide a more complete measurement as they acquire a large number of measurement points simultaneously. Fringe projection techniques 4,5 are a category of SLS where either continuous 6-8 (usually sinusoidal) or binary fringes 6,9,10 are projected on an object and images are taken at an angle in order to extract three-dimensional (3-D) topography information about the object. For the fringe projection technique to operate efficiently and for the system to have adequate depth resolution, the camera and projector must be noncollinear. [11][12][13] In this work, we do not consider techniques that use collinear fringe projection 12,13 because as they do not employ standard phase stepping methods. In this work, we will focus on systems that use sinusoidal fringe projection and phase stepping. 14 The noncollinearity criterion in fringe projection is a significant limitation when measuring complex objects with self-occlusions, such as those produced by additive manufacturing (AM) techniques. 2,[15][16][17] During the measurement of complex objects, self-occlusions of the object geometry do not allow the camera to view all illuminated parts of the object. To acquire a full 3-D measurement of the object, the general solution is to place the object on a stage and rotate it multiple times with small angular increments to acquire a complete measurement, however, this approach is inefficient and time consuming. Another common solution to overcome object self-occlusions is to add extra cameras around the projector, but this increases the cost and the complexity of the system. Moreover, the use of additional cameras and projectors is not an ideal solution as the distances and relative poses between the projector and cameras are still static and cannot easily adapt to different objects. The algorithm presented in Ref. 18 has been shown to be tolerant to the arbitrary relative position and pose between camera and projector, but it requires calibration between each alteration and is thus time consuming.
In this work, we propose a system design capable of independent camera and projector motion through live tracking of the camera. Moreover, we show that the proposed system is more capable to overcome self-occlusions compared to traditional systems, where the camera is not free to move independently of the projector.

Fringe Projection Procedure
The principle used in phase stepped fringe projection is the association of the absolute phase measured by the process to a physical height. To perform a coordinate measurement in fringe projection, at least three-phase stepped images of the projected pattern are acquired by the camera. Equations (1)-(3) 19 describe the intensity of the sinusoidal pattern detected by the camera pixels in each image in the case of a three-step measurement E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 3 2 6 ; 2 2 5 I 1 ðx; yÞ ¼ I 0 ðx; yÞ þ I 00 ðx; yÞ cos½φðx; yÞ − Δφ; (1) E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 3 2 6 ; 1 9 1 I 2 ðx; yÞ ¼ I 0 ðx; yÞ þ I 00 ðx; yÞ cos½φðx; yÞ; (2) E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 3 2 6 ; 1 6 2 I 3 ðx; yÞ ¼ I 0 ðx; yÞ þ I 00 ðx; yÞ cos½φðx; yÞ þ Δφ; where x and y are the pixel coordinates, I 1 − I 3 are the pixel intensities recorded in the images, I 0 is the background illumination intensity, I 00 is the modulation intensity, φ is the wrapped phase of the sinusoidal pattern in the second image, and Δφ is the phase step, which in the case of a three-step acquisition is 2π∕3. By solving the system of equations above, we can extract the relative phase in terms of the intensity in each image 19 thus E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 6 3 ; 4 8 7 φðx; yÞ ¼ arctan The phase is then unwrapped 20 and converted into dimensional units by scaling the phase with an object of known height.

Occlusions Created by High Aspect Ratio Features
Figure 1(a) shows the geometrical constraints in measuring trenches with high aspect ratios when a camera and projector system have a relatively large angle between them. In this case, no overlap between the projector and camera pixels is possible at the bottom of the trench and, therefore, the depth of the trench cannot be measured. Deep trench self-occlusions can be overcome if the camera and object move independently of each other and adapt to the object being measured by moving closer together [ Fig. 1(b)]. However, when the angular distance between the camera and projector is small, the system is less able to determine the phase difference between the projected and detected image. 11 The optimal case for the measurement of such self-occlusions is, therefore, to keep the maximum angle difference between the camera and projector while still being able to view all the necessary parts of the object. An additional advantage of being able to achieve independent angular motion of the camera with respect to the projector would be the acquisition of two sides of an illuminated ridge self-occlusion (see Fig. 2), a problem usually solved by registering the views of two independent cameras.
A flexible system that allows the camera-projector angles to vary both in azimuth and elevation would have the ability to minimize the effect of such self-occlusions and acquire almost all of the illuminated area of the object, thus making the measurement of a 3-D object more efficient. Such a system would also be able to perform automatic point cloud registration, whereby 3-D point clouds acquired from different parts of the object can be registered to a common coordinate system via use of inertial sensors. The use of inertial sensors for automatic point registration has recently been shown, 21 but the sensor was not placed on the camera itself, rather on the projector-camera frame, which was static, and the camera and projector did not move relative to each other.

Camera Calibration
The camera and projector cannot move independently to each other during the measurement due to the calibration procedure used in existing systems. Camera calibration for metrological applications is usually performed by well-established methods, such as linear calibration, 22 the two-stage Tsai method, 22,23 vanishing points, 24 and the checkerboard plane technique (Zhang method). 22,25 All these techniques are used to determine the matrix P shown in E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 5 ; 3 2 6 ; 2 9 8 where w is the scale factor; x; y are the coordinates of the image along the horizontal and vertical directions, respectively; X; Y, and Z are the spatial coordinates of the corresponding pixel in the real world; and P is known as the projection matrix. The P matrix contains both the extrinsic and intrinsic parameters of the camera. The intrinsic parameters of the camera are associated with the exact values of the internal camera parameters, such as the focal length, the optical centre, and the pixel size. The extrinsic parameters are associated with the location and pose of the camera with respect to a reference coordinate system, also referred to as the "global coordinates." When using the aforementioned methods to calibrate the camera, the camera is unable to move with respect to the projector during the calibration procedure, as this would require the whole system to be recalibrated. The reason for this is that, when moving the camera around an object, the extrinsic parameters vary as does the P matrix in Eq. (1), which has to be recalculated. The novelty of this work is that we first calibrate the intrinsic and extrinsic parameters with the Zhang method and then use inertial measurement unit (IMU) sensors to determine the change in extrinsic parameters when moving the camera, thus avoiding the need to recalibrate.

Sensor Calibration
In order to use the angular data of the IMU sensor (LSM9DS0), it must first be calibrated. The variation of the azimuth angle sensor (which uses a magnetometer to acquire the data reported) with respect to the true rotation is initially very large. The calibration of the magnetometer sensor was carried out in a piecewise linear fashion via a look-up table at an interval of 30 deg. The true azimuth rotation was measured by the use of a rotation stage with angular markings. Any reported angular value from the sensor can be acquired by performing a linear interpolation between two appropriate calibration points. The deviation reported before calibration was on average 32.9 deg. After calibration, when the difference was measured at random angles within each linearly approximated region, the average error reported was 1.49 deg.
The tilt angle sensor in the IMU is an accelerometer and its response was linear when measured in the 0-deg to 90-deg range. The tilt angle was calibrated with a digital protractor with an accuracy of 0.1 deg. As the response of the tilt sensor was linear, the results were corrected by subtracting the error of the reported angle at the centre of the measured range (at a tilt of 45 deg). The average error before calibration was 3.6 deg and after calibration, when measured at different points, was confirmed to be 0.6 deg.

Experiments
A schematic and photos of the experimental setup used are shown in Fig. 3. The design used ensures that the distance of the camera to the object does not change as it is rotated around the object. Keeping the radial distance invariant as the camera rotates around the object, camera tracking and point cloud registration can be performed by sole use of the azimuth and elevation angles, without requiring knowledge of the distance to the object. In this setup, the camera can be placed at five different elevation angles on each side of the arch by physically detaching and attaching the camera through threaded screws at each position.
The camera used is a modified Raspberry Pi camera with 5-megapixel resolution. The camera was attached to the front of a custom-made case. The Raspberry Pi board used was connected to the network via an USB Wi-Fi dongle.
The setup was evaluated using two AM manufactured objects printed in acrylonitrile butadiene styrene. The two objects were made to have different types of self-occlusions. The first was a pyramid with a tall ridge self-occlusion [ Fig. 4(a)] and the other was a flat object with deep trench self-occlusions [ Fig. 4(b)].
The aim of the first experiment was to demonstrate the system's ability to avoid the pyramid's self-occlusions and acquire a larger part of the illuminated object. Two point cloud measurements of the AM manufactured pyramid object [ Fig. 4(a)] were performed by moving the camera in two different positions while keeping the projector static. The two views were spaced angularly by ∼85 deg. The camera's elevation angle in both cases was 30 deg from normal. By acquiring two different views while keeping the same projection illumination direction, more of the illuminated field was captured. The two point clouds were subsequently aligned to the computer aided design (CAD) model of the object via an iterative closest point (ICP) using CloudCompare (CC) software. 26 The point clouds of the two views and the combined registration via ICP are shown in Fig. 5.
To demonstrate the advantage gained when measuring an object manufactured with deep trenches, the AM object [ Fig. 4(b)] was also measured at two different viewpoints with the setup proposed. The point clouds acquired from two different azimuth angles (Fig. 6) clearly show the ability of the system to measure the points in the bottom of the trench after rotating the camera closer to the projector. The camera's elevation angle in both cases was set to 30 deg from normal.

Discussion and Conclusion
It was found that the use of a magnetometer sensor for azimuth angle tracking can cause large errors, up to AE30 deg, in the measurement when uncalibrated. The specific reasons for the high nonlinearity exhibited by the uncalibrated magnetometer was not investigated, nonetheless, with the calibration techniques used in this work we were able to reach accuracies of 1.49 deg for the azimuth measurement after calibration. The tilt sensor albeit essentially not used for tracking in this case, as all shots were taken at the same tilt angle, is an accelerometer sensor and was linear over the 0-deg to 90-deg range over which we tested it and could be calibrated by a simple offset. Specifically, we were able to achieve an average error of 0.6 deg for the camera tilt after calibration.
The calculated value of the movement in azimuth angle, when measuring two different parts of the AM pyramid object after ICP registration of the two point clouds acquired, was 86.75 deg. When comparing with the data acquired by the IMU sensor for the same motion (85.72 deg), a difference of 1.03 deg was observed. This difference is within the expected mean sensor error of 1.49 deg, which was achieved for the azimuth sensor after calibration. The mean difference between the ICP registered and IMU tracked point clouds, which resulted from the 1.03-deg azimuth sensor error, was measured to be 18 μm. The average point cloud to the CAD model error resulting from the sensor errors is expected to vary with the type and size of the object as the angular error would affect different objects differently. The detailed effect of the angular errors on registration accuracy was not the focus of this work and requires a more in-depth investigation, which is going to be part of future work.
With regards to the enhanced measurement of deep trenches using the system, it is clearly shown that by varying the angle of the camera closer to that of the projector, it is possible to acquire points at the bottom of the trenches of the AM object evaluated. In terms of the numerical differences of the point cloud distributions, these were acquired by initially registering them to the CAD model individually and then finding the mean distance of the less dense point cloud [ Fig. 6(a)] from the denser one [ Fig. 6(b)]. The point cloud of the deep trench artefact, which was taken when the camera was at a smaller angle to the projector [ Fig. 6(b)], (i.e., in which the trenches were clearly visible), showed a mean error that was reduced by 59 μm compared to the point cloud that was taken at a larger angle [ Fig. 6(a)]. The reason for this is not clear, but it was noticed that the point cloud in Fig. 6(a) was noisier compared to that in Fig. 6(b), especially in the areas close to the trench edges. Again, the detailed analysis of the errors warrants further investigation but the advantage provided in measuring deep trenches by the system is clear.
In both cases, the average differences in mean error between the point clouds were presented for the sake of completeness. The focus of this work was to demonstrate the advantages and the first realisation of a fringe projection system where the camera and projector can move independently during the measurement by use of inertial sensor tracking. The main advantage demonstrated is to allow complete 3-D measurement of objects with different types of high aspect ratio occlusions by adapting the setup during the course of the measurement process. Further investigation of the detailed effects of sensor errors on the point cloud accuracies and identifying other sources of error in the setup, as well as methods of reducing or mitigating them, will be part of future work.
This work, therefore, demonstrates that, for a calibrated fringe projection setup in which the camera can move radially around the object, the camera's extrinsic parameters can be determined solely by the measurement of the two pose angles (i.e., the camera's elevation and azimuth). The ability to calibrate the extrinsic parameters of a camera in real-time  using data from an inertial sensor allows for the camera to be moved around the object and acquire a larger part of the illuminated area. Additionally, each image taken from a different point of view can be "tagged" with the position of the camera's pose making the time-consuming task of point cloud registration more streamlined and automated. For setups where the distance of the camera to the object changes, however, the elevation and azimuth angles are not enough to track the position of the camera, as the distance of the camera to the object has to be measured in order to properly adjust the scale of the image, and the method shown here cannot be used.
It has been further shown that using the proposed setup, it is possible to minimize the effect of object self-occlusion in complex geometries using two examples. Measuring complex geometries with self-occlusions is important for AM objects that can produce a lot of lattice and biomimetic structures, which can contain deep-trench and tall-ridge selfocclusions. It was shown that point cloud registration to within 1 deg in azimuth is possible (compared to ICP) via sole use of the inertial sensors and without the need to use specific targets or multiple cameras.
A postprocessing algorithm, such as ICP, is still recommended to fully register the point clouds if alignment accuracy is paramount. However, if the highest possible alignment accuracy is not required, or if coarse initial alignment is needed, using the calibrated inertial sensors in the setup shown also allows for adequate registration with an angular error of 1 deg in azimuth, in which the specific pyramid artefact measured, translated to a mean point cloud displacement error of 18 μm.
The scope of the future work, aside from creating a framework to fully optimize the angle selection process between the projector and camera for different objects and characterizing the effect of the inertial sensor errors reported, would be to employ a two-step registration scheme whereby the point clouds collected would be initially registered through inertial sensing and, subsequently, through ICP to allow for a faster and fully automated point cloud registration process. Future work will also concentrate on creating a setup that allows automated motion of the camera in the elevation axis and, therefore, can also demonstrate the mitigation of self-occlusions, which are vertical to the reference measurement plane.