1.

Introduction

Structured light systems (SLS)^1,2 are used to perform dimensional measurements on objects by projecting patterns of light, which are subsequently recorded by a camera. SLS have recently proliferated in industrial metrology, because they do not contact the object being measured and thus can be used with final-stage products; they operate much faster than standard contact coordinate measuring machines³ and provide a more complete measurement as they acquire a large number of measurement points simultaneously. Fringe projection techniques^4,5 are a category of SLS where either continuous^6–8 (usually sinusoidal) or binary fringes^6,9,10 are projected on an object and images are taken at an angle in order to extract three-dimensional (3-D) topography information about the object. For the fringe projection technique to operate efficiently and for the system to have adequate depth resolution, the camera and projector must be noncollinear.^11–13 In this work, we do not consider techniques that use collinear fringe projection^12,13 because as they do not employ standard phase stepping methods. In this work, we will focus on systems that use sinusoidal fringe projection and phase stepping.¹⁴

The noncollinearity criterion in fringe projection is a significant limitation when measuring complex objects with self-occlusions, such as those produced by additive manufacturing (AM) techniques.^2,15–17 During the measurement of complex objects, self-occlusions of the object geometry do not allow the camera to view all illuminated parts of the object. To acquire a full 3-D measurement of the object, the general solution is to place the object on a stage and rotate it multiple times with small angular increments to acquire a complete measurement, however, this approach is inefficient and time consuming. Another common solution to overcome object self-occlusions is to add extra cameras around the projector, but this increases the cost and the complexity of the system. Moreover, the use of additional cameras and projectors is not an ideal solution as the distances and relative poses between the projector and cameras are still static and cannot easily adapt to different objects. The algorithm presented in Ref. 18 has been shown to be tolerant to the arbitrary relative position and pose between camera and projector, but it requires calibration between each alteration and is thus time consuming.

In this work, we propose a system design capable of independent camera and projector motion through live tracking of the camera. Moreover, we show that the proposed system is more capable to overcome self-occlusions compared to traditional systems, where the camera is not free to move independently of the projector.

2.

Background

2.2.

Occlusions Created by High Aspect Ratio Features

Figure 1(a) shows the geometrical constraints in measuring trenches with high aspect ratios when a camera and projector system have a relatively large angle between them. In this case, no overlap between the projector and camera pixels is possible at the bottom of the trench and, therefore, the depth of the trench cannot be measured.

Fig. 1

Demonstration of a deep trench self-occlusion which could be measured if the angle between camera and projector is reduced.

Deep trench self-occlusions can be overcome if the camera and object move independently of each other and adapt to the object being measured by moving closer together [Fig. 1(b)]. However, when the angular distance between the camera and projector is small, the system is less able to determine the phase difference between the projected and detected image.¹¹ The optimal case for the measurement of such self-occlusions is, therefore, to keep the maximum angle difference between the camera and projector while still being able to view all the necessary parts of the object.

An additional advantage of being able to achieve independent angular motion of the camera with respect to the projector would be the acquisition of two sides of an illuminated ridge self-occlusion (see Fig. 2), a problem usually solved by registering the views of two independent cameras.

Fig. 2

Demonstration of the ability to measure both sides of a ridge self-occlusion by moving the camera to the other side of the projector.

A flexible system that allows the camera–projector angles to vary both in azimuth and elevation would have the ability to minimize the effect of such self-occlusions and acquire almost all of the illuminated area of the object, thus making the measurement of a 3-D object more efficient. Such a system would also be able to perform automatic point cloud registration, whereby 3-D point clouds acquired from different parts of the object can be registered to a common coordinate system via use of inertial sensors. The use of inertial sensors for automatic point registration has recently been shown,²¹ but the sensor was not placed on the camera itself, rather on the projector–camera frame, which was static, and the camera and projector did not move relative to each other.

3.

Calibration

4.

Experiments

A schematic and photos of the experimental setup used are shown in Fig. 3. The design used ensures that the distance of the camera to the object does not change as it is rotated around the object. Keeping the radial distance invariant as the camera rotates around the object, camera tracking and point cloud registration can be performed by sole use of the azimuth and elevation angles, without requiring knowledge of the distance to the object. In this setup, the camera can be placed at five different elevation angles on each side of the arch by physically detaching and attaching the camera through threaded screws at each position.

Fig. 3

(a) A schematic of the setup showing the elevation ($\theta$) and azimuth ($\phi$) angles as well as the distance of the camera to the centre of the rotation plate ($R$) and images of the Raspberry Pi camera setup showing (b) the camera under the arch and (c) a full shot of the rotating setup on an optical wafer.

The camera used is a modified Raspberry Pi camera with 5-megapixel resolution. The camera was attached to the front of a custom-made case. The Raspberry Pi board used was connected to the network via an USB Wi-Fi dongle.

The setup was evaluated using two AM manufactured objects printed in acrylonitrile butadiene styrene. The two objects were made to have different types of self-occlusions. The first was a pyramid with a tall ridge self-occlusion [Fig. 4(a)] and the other was a flat object with deep trench self-occlusions [Fig. 4(b)].

Fig. 4

(a) Pyramid-shaped and (b) deep trench 3-D printed objects tested with the flexible fringe projection setup. The size of the pyramid was $10\mathrm{cm}\times 10\mathrm{cm}\times 7\mathrm{cm}$ and the size of the trench object is $10\mathrm{cm}\times 10\mathrm{cm}\times 2\mathrm{cm}$.

The aim of the first experiment was to demonstrate the system’s ability to avoid the pyramid’s self-occlusions and acquire a larger part of the illuminated object. Two point cloud measurements of the AM manufactured pyramid object [Fig. 4(a)] were performed by moving the camera in two different positions while keeping the projector static. The two views were spaced angularly by $\sim 85\mathrm{deg}$. The camera’s elevation angle in both cases was 30 deg from normal. By acquiring two different views while keeping the same projection illumination direction, more of the illuminated field was captured. The two point clouds were subsequently aligned to the computer aided design (CAD) model of the object via an iterative closest point (ICP) using CloudCompare (CC) software.²⁶ The point clouds of the two views and the combined registration via ICP are shown in Fig. 5.

Fig. 5

ICP registered point clouds showing the data portion acquired from the two individual camera views as seen from the point of view of the projector. (a) The camera is situated at the left of the projector. (b) The camera at the right of the projector. (c) The combined data acquired from both views overlaid onto the object’s CAD model.

To demonstrate the advantage gained when measuring an object manufactured with deep trenches, the AM object [Fig. 4(b)] was also measured at two different viewpoints with the setup proposed. The point clouds acquired from two different azimuth angles (Fig. 6) clearly show the ability of the system to measure the points in the bottom of the trench after rotating the camera closer to the projector. The camera’s elevation angle in both cases was set to 30 deg from normal.

Fig. 6

(a) Point cloud of an object with deep trenches, where the pits could not be measured due to the relative projector and camera angle (53.3 deg). (b) By changing the camera angle to 16.8 deg, the bottom parts of the trenches were successfully acquired the trench width and the separation between trenches was 1 cm. The size of the AM object is $10\mathrm{cm}\times 10\mathrm{cm}\times 2\mathrm{cm}$.

5.

Discussion and Conclusion

It was found that the use of a magnetometer sensor for azimuth angle tracking can cause large errors, up to $\pm 30\mathrm{deg}$, in the measurement when uncalibrated. The specific reasons for the high nonlinearity exhibited by the uncalibrated magnetometer was not investigated, nonetheless, with the calibration techniques used in this work we were able to reach accuracies of 1.49 deg for the azimuth measurement after calibration. The tilt sensor albeit essentially not used for tracking in this case, as all shots were taken at the same tilt angle, is an accelerometer sensor and was linear over the 0-deg to 90-deg range over which we tested it and could be calibrated by a simple offset. Specifically, we were able to achieve an average error of 0.6 deg for the camera tilt after calibration.

The calculated value of the movement in azimuth angle, when measuring two different parts of the AM pyramid object after ICP registration of the two point clouds acquired, was 86.75 deg. When comparing with the data acquired by the IMU sensor for the same motion (85.72 deg), a difference of 1.03 deg was observed. This difference is within the expected mean sensor error of 1.49 deg, which was achieved for the azimuth sensor after calibration. The mean difference between the ICP registered and IMU tracked point clouds, which resulted from the 1.03-deg azimuth sensor error, was measured to be $18\mu \mathrm{m}$. The average point cloud to the CAD model error resulting from the sensor errors is expected to vary with the type and size of the object as the angular error would affect different objects differently. The detailed effect of the angular errors on registration accuracy was not the focus of this work and requires a more in-depth investigation, which is going to be part of future work.

With regards to the enhanced measurement of deep trenches using the system, it is clearly shown that by varying the angle of the camera closer to that of the projector, it is possible to acquire points at the bottom of the trenches of the AM object evaluated. In terms of the numerical differences of the point cloud distributions, these were acquired by initially registering them to the CAD model individually and then finding the mean distance of the less dense point cloud [Fig. 6(a)] from the denser one [Fig. 6(b)]. The point cloud of the deep trench artefact, which was taken when the camera was at a smaller angle to the projector [Fig. 6(b)], (i.e., in which the trenches were clearly visible), showed a mean error that was reduced by $59\mu \mathrm{m}$ compared to the point cloud that was taken at a larger angle [Fig. 6(a)]. The reason for this is not clear, but it was noticed that the point cloud in Fig. 6(a) was noisier compared to that in Fig. 6(b), especially in the areas close to the trench edges. Again, the detailed analysis of the errors warrants further investigation but the advantage provided in measuring deep trenches by the system is clear.

In both cases, the average differences in mean error between the point clouds were presented for the sake of completeness. The focus of this work was to demonstrate the advantages and the first realisation of a fringe projection system where the camera and projector can move independently during the measurement by use of inertial sensor tracking. The main advantage demonstrated is to allow complete 3-D measurement of objects with different types of high aspect ratio occlusions by adapting the setup during the course of the measurement process. Further investigation of the detailed effects of sensor errors on the point cloud accuracies and identifying other sources of error in the setup, as well as methods of reducing or mitigating them, will be part of future work.

This work, therefore, demonstrates that, for a calibrated fringe projection setup in which the camera can move radially around the object, the camera’s extrinsic parameters can be determined solely by the measurement of the two pose angles (i.e., the camera’s elevation and azimuth). The ability to calibrate the extrinsic parameters of a camera in real-time using data from an inertial sensor allows for the camera to be moved around the object and acquire a larger part of the illuminated area. Additionally, each image taken from a different point of view can be “tagged” with the position of the camera’s pose making the time-consuming task of point cloud registration more streamlined and automated. For setups where the distance of the camera to the object changes, however, the elevation and azimuth angles are not enough to track the position of the camera, as the distance of the camera to the object has to be measured in order to properly adjust the scale of the image, and the method shown here cannot be used.

It has been further shown that using the proposed setup, it is possible to minimize the effect of object self-occlusion in complex geometries using two examples. Measuring complex geometries with self-occlusions is important for AM objects that can produce a lot of lattice and biomimetic structures, which can contain deep-trench and tall-ridge self-occlusions. It was shown that point cloud registration to within 1 deg in azimuth is possible (compared to ICP) via sole use of the inertial sensors and without the need to use specific targets or multiple cameras.

A postprocessing algorithm, such as ICP, is still recommended to fully register the point clouds if alignment accuracy is paramount. However, if the highest possible alignment accuracy is not required, or if coarse initial alignment is needed, using the calibrated inertial sensors in the setup shown also allows for adequate registration with an angular error of 1 deg in azimuth, in which the specific pyramid artefact measured, translated to a mean point cloud displacement error of $18\mu \mathrm{m}$.

The scope of the future work, aside from creating a framework to fully optimize the angle selection process between the projector and camera for different objects and characterizing the effect of the inertial sensor errors reported, would be to employ a two-step registration scheme whereby the point clouds collected would be initially registered through inertial sensing and, subsequently, through ICP to allow for a faster and fully automated point cloud registration process. Future work will also concentrate on creating a setup that allows automated motion of the camera in the elevation axis and, therefore, can also demonstrate the mitigation of self-occlusions, which are vertical to the reference measurement plane.