Predesign of a projector alignment system for structured light illumination systems

Abstract. Structured light illumination systems can measure a target’s two-dimensional height profile. Recent advancements in computational power and algorithms have enabled the predesign of optical systems where all experimental variables are determined prior to use in the laboratory. We present a predesign process to align the projector of a custom-structured light illumination system. The predesign method uses the native optical design file to simulate the residual aberrations and can perform a trade space analysis to tailor the alignment variables to desired preferences. The presented method can predict the experimental focus position to 1.6  μm root mean square over a broad range of physical settings.


Overview
Determining the height profile of a target is useful in many applications, such as industrial production where rapid feature recognition is key to high fidelity and low cost. One common noncontact method to produce height profiles uses structured light illumination (SLI), 1,2 in which a pattern of light from a projector illuminates the object and camera images the scene. The pattern varies spatially and is further distorted by the sample's three-dimensional (3D) shape. Image processing techniques then reconstruct the 3D image and produce the desired height maps. 3 As with most optical systems, accurate optical alignment of SLI systems is key to the reconstruction resolution and performance.
Recent advances in computer power and algorithms have enabled the predesign of optical systems. Côté et al. 4 created a deep-learning enabled framework called LensNet that extracts features of successful lens designs and then recombines the features to create new layouts. Ivanov et al. 5 published a method to determine the number of optical surfaces necessary to achieve diffraction-limited performance prior to any actual optical design. Oleszko and Gross 6 used a numerical model to study freeform optical systems surface-by-surface and correct for aberrations beyond the fourth-order. This paper applies the predesign process to projector alignment in custom SLI systems. The projector is aligned with the quad target method (QTM), 7 a simple height reconstruction technique that produces an estimate of the projector's focal position using only a single camera image. The presented predesign method uses the native optical design file to include the effect of residual aberrations, a key step for accurate simulation of the experimental behavior. The goal here is to quickly move from the design to the lab by enabling a user to completely design the alignment process variables prior to any lab work, even tailoring the responses with a trade space analysis. The simulated alignment process is compared with experimental measurements on a custom SLI system to determine the accuracy of the predesign method. Results suggest that the predesign method can accurately model the experimental behavior. *Address all correspondence to Matthew E. L. Jungwirth, mejungwirth@gmail.com This paper is organized into five sections. The remainder of Sec. 1 briefly describes SLI and the QTM. Then, Sec. 2 displays the test system since the optical design must be well-established prior to utilizing the presented predesign method. Section 3 presents the predesign process including the coordinate system and a simulation flowchart. Section 4 describes the experimental methods and displays and compares the experimental and simulated results. Conclusions are found in Sec. 5.

Structured Light Illumination
One common method to perform 3D profilometry is structured light illumination (SLI). A schematic of a general SLI system is shown in Fig. 1. 2 Here, the structured light projector images a particular pattern onto the target. Many patterns are found in the literature, such as binary, 8 graylevel, 9 and colored bar patterns. 10 A popular subset of SLI is fringe projection profilometry (FPP) where the projected pattern is sinusoidal fringes. [11][12][13] The rectangular target distorts the color bars in Fig. 1 due to its height above the substrate. A camera then images the entire scene. The combination of the camera image and a processing method enables the reconstruction of the target's 3D shape. 1,2

Quad Target Method
This paper utilizes the quad target method (QTM), 7 an alignment technique that features a linearized response to focus the projector of an SLI or FPP system. The linearization is achieved by sampling the field at two different locations along the optical axis using the quad target. The quad target is a 2 × 2 array of flat surfaces in pairs of quadrants separated by a step height h step as seen in Fig. 2. Then, focus is efficiently linearized by measuring the difference in the projected fringe frequency contrasts between the two quad target heights. QTM can produce an estimate of the focus position using only a single image, a 3× improvement in speed over the common threephase reconstruction technique. 3 Thus, QTM provides linear feedback with a clear zero-crossing, to minimize any focus ambiguity, and fast positional feedback, to minimize adjustment lag.
QTM is governed by two input variables: the frequency of the projected fringes ξ project and the height of the quad target step h step ; it outputs two quantities: the slope of the linearized values m L and an estimate of the current focus position Z focus . A schematic of QTM is shown in Fig. 2. Two field points (red and blue) of sinusoidal fringes at frequency ξ project are imaged by the projector optics onto the quad target, a 2 × 2 array of flat surfaces physically separated along the optical axis, with two of the four steps shown in profile in Fig. 4. Notice that due to the quad target step height h step (greatly exaggerated for viewing ease), the blue field is in focus and the red defocused. A camera images the entire scene with the camera's front working distance labeled Z 1 . The distance Z 2 is the projector's back working distance which can be physically altered to shift the projector's focus position during alignment.  shows an example output of QTM. The raw and fitted fringe contrast values from the front step are in blue with the same curves for the back step are in red. The raw data is fit to a fifth-order polynomial (dotted lines) for noise reduction and robustness with a curve shape. Note that in this paper the fitted data curves for the front and back steps are called the F and B curves, respectively. Next, the linearized response L (orange asterisk) is calculated from the F and B curves and also fit to fifth-order polynomial (orange dotted line) for the same reasons. Then, the nonlinear coefficients of the fit are set to zero to ensure that L is linear near the calibrated focus position at Z ¼ 0. Finally, the black dotted line shows the current focus position of Z ¼ 6.2 μm, i.e., where L ¼ 0 by definition of QTM. 7 As stated above, the raw F; B, and L curves in Fig. 3 are all individually fit to fifth-order polynomials. For the given input variables, the curves in Fig. 3 are well-approximated by a  Notice that the blue field is focused on the front step and the red field is defocused on the back step. Other aspects: the thick black line is the optical axis, the digital micromirror device (DMD) produces the SLI pattern and is tilted to meet the Scheimpflug condition, distance Z 1 is the camera's target-space working distance, and distance Z 2 is the projector's back working distance. 7 second-order polynomial leading to the potential for errors caused by over-fitting. However, Ref. 7 showed that fifth-order fits do not lead to over-fitting for a wide range of QTM input variables. Additionally, proper choice of the projected light level and the large averaging described in Sec. 3.1 should ensure no over-fitting of the curves.

Layout
This paper begins with the test system, since the design must be established before the alignment method can be predesigned. The custom SLI sensor 14 shown in Fig. 4 has two cameras and four projectors, meaning that there are eight total SLI channels. Throughout this paper, however, only a single projector/camera combination was examined for brevity. A listing of the optical attributes of the sensor is found in Table 1. Note that the optical design is outside the scope of this paper.
In Table 1, two fields of view (FOVs) are listed: 3D and raw. The 3D FOV is the lateral region where the sensor can measure and generate 3D height maps, whereas the raw FOV is the full FOV of the camera. The raw FOV is slightly larger than the alignment FOV to ensure good alignment across the 3D FOV and to reduce edge effects during height reconstruction.

Through Focus Performance
As will be described in Sec. 4, an important aspect in predicting the alignment performance is the through-focus response of the test system. Figure 5 shows the through-focus modulation transfer function (MTF) contrasts for three spatial frequencies using the native optical design.
These specific frequencies will be tested in Sec. 4. Notice that the highest spatial frequency of 13.7 cy/mm in Fig. 5(a) has a much lower depth of field than the lowest frequency of 3.4 cy/mm in (c), and that the contrast dynamic range for 13.7 cy/mm is much larger than 3.4 cy/mm (where "dynamic range" is the difference between the maximum and minimum values). Thus, the tradeoff with ξ project is higher spatial frequencies resulting in more sensitivity but narrower Z range for adjustment.

Coordinate System
The alignment predesign is performed via simulation. The coordinate system for the simulation method is shown in Fig. 6. The alignment FOV (red box) is slightly larger than the system FOV (green box), as described in Sec. 2.1. The black boxes show the experimental regions of interest (ROIs), i.e., the ROIs utilized during the projector alignment. The experimental ROIs are set as large as possible to reduce noise while avoiding the edges of the quad target step. To match the experimental ROIs as much as possible, the simulated field points are placed at the center of the four ROIs, thereby assuming that the aberrations across the ROI are roughly balanced at the center.

Simulation Flowchart
The simulation method uses the native optical design file. With this approach, the residual aberrations are accurately modeled, and their effects are included in the predicted response. Therefore, simulating with the native file is a key step for accurate simulation of the experimental behavior.
The simulation raytracing was performed in Zemax OpticStudio; 15 using the ZOS-API, a MATLAB 16 script managed the data flow and graphed the results. Figure 7 shows a flowchart of the simulation code. The QTM variables of h step and ξ project are set to the desired values. Then, the target plane is shifted by AEh step to simulate the front and back surfaces of the quad target, Fig. 6 Layout of coordinate plane for predesign and experiment: blue asterisks are the simulated field points; black boxes are the experimental regions of interest; the green box is the test system's field of view; the red box is the alignment field of view. Fig. 7 Flowchart for simulation of the test system using OpticStudio 15 and MATLAB. 16 The steps with blue, dotted lines were performed in Zemax, while the black outlines were in MATLAB.
Jungwirth: Predesign of a projector alignment system for structured light illumination systems Optical Engineering 021003-6 February 2023 • Vol. 62 (2) respectively. Next, the contrasts are determined for the four field points shown in Fig. 6 using the operand MTFT since the triangulation axis and tangential axis are parallel. The F and B contrasts are determined by multiplying the contrasts from the projector and camera, 17 specifically E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 1 1 6 ; 6 9 9 CðR n Þ ¼ C project ðh step ; ξ project Þ · C camera ðh step ; ξ project Þ; (1) where the C project and C camera are the contrasts of the projector and camera, respectively. Finally, the linearized value L is determined via QTM. Distortion is of particular concern to SLI systems because it causes the projected fringe frequency to change across the FOV and, in turn, alters the measured fringe contrast. Since the native optical design file is used in the simulation method, the distortion's effect is already included in the prediction; therefore, there is no need to correct the camera's distortion for the presented simulation method. If additional distortion is found experimentally, the unmodeled distortion can be mitigated by decreasing the ROIs in Fig. 6 to avoid the FOV edges where distortion is at its largest magnitude, or by increasing the width of the bandpass filter used to determine the contrast to ensure that the altered frequency is measured.

Simulation Outputs
With this method, the test system's response can be simulated. Figure 8 displays the F; B, and L curves through focus for h step ¼ 500 μm and ξ project ¼ 6.9cy∕mm. This graph shows the Z position along the X axis; the normalized contrast on the left-hand Y axis; and the linearized value L on the right-hand Y axis. The main two outputs of QTM are garnered from Fig. 8: the slope of the linearized values m L and the current projector focus position Z focus . The slope m L affects the resolution and "feel" of the adjustment for the user.
The current focus position Z focus is estimated using the linear fit of L. By definition of QTM, the current focus position is where L ¼ 0, so the focus position Z focus is estimated as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 1 1 6 ; 4 1 3 where m L and b are the coefficients for linear slope and bias, respectively.

Trade Space Analysis
We can probe the effect of h step and ξ project with a large-scale trade space analysis. 18 Here, both variables are altered and the predesign method determines m L and Z focus . This process has a unique benefit for the user: it allows the user to tailor the alignment method for their preferences. For example, if a user desires maximum adjustment sensitivity, the user might employ the highest projected fringe frequency to maximize m L .

Experimental Process
To test the accuracy of the predesign method presented in Sec. 3, experimental measurements of m L and Z focus were taken using the test system in Sec. 2. As shown in Table 2, seven different combinations of step heights and spatial frequencies were examined to sample the two variables across a wide range. Specifically, h step covers 1×, 3×, and 5× of the camera's depth of field and ξ project covers 0.24×, 0.49×, 1× of the projector's maximum spatial frequency (referring to Table 1). Notice that combinations of low ξ project and low and mid h step are not included since Ref. 7 showed that these variable combinations do not produce accurate estimates of focus position over Z ranges of ∼1000 μm. Figure 9 shows two of the three tested quad targets resting below the test sensor. The targets are displaced in Z to change the distance Z 1 in Fig. 2. For all experimental and simulated results Table 2 Table of h step and ξ project combinations tested both experimentally and in simulation.

Run #
Step height (μm) Spatial frequency (cy/mm) in Sec. 7, the range of Z values was AE500 μm with a 50 μm step size. The stage is a Newport GTS30V and is controlled via a MATLAB script. Finally, the test sensor is rigidly mounted to a granite frame to reduce vibrations, as seen in Fig. 4.

Single Comparison
As an initial comparison between the experiment and the predesign method, Fig. 10 shows the experimental curves from Run #3 with the simulated curves from Fig. 8. The error bars in Fig. 10 are AE2σ from four experimental trials. Overall, the predesign method seems to predict the experimental curves well, as evidenced by the good visual agreement and high R 2 values in Table 3. Figure 10 also shows the most straightforward method to demonstrate the simulated and experimental agreement since it only uses one set of input variables.

Trade Space Analysis
As described in Sec. 3.3, the predesign method can perform a trade space analysis of the variable space. The analysis studied ξ project and h step over 5× ranges, with ξ project starting from the maximum fringe frequency and decreasing 5×, and h step starting from 1× the camera's depth of field and increasing. Then, to further examine the predesign method's accuracy, we compared the experimental and simulated values for m L and Z focus .

Linearized slope
One of the two outputs of QTM is the slope of the L curve m L . Figure 11(a) shows the simulated slope m L and (b) the localized gradient of the surface in (a). The mean angle of the gradient is 59.0 deg meaning that ξ project has a slightly stronger effect on m L than h step . Thus, if a user desires the alignment to be more sensitive, i.e., a higher m L , increasing the fringe frequency or step height will have roughly the same effect. The seven combinations in Table 2 were tested experimentally using the test system in Sec. 2. The results are displayed graphically in Fig. 12 with summary statistics in Table 4. The graph exhibits very good linear regression (the dotted line) both visually and with R 2 ¼ 0.99. Furthermore, the root mean square (RMS) difference for all runs is 0.02 mm −1 and the regressed  slope are very close to ideal at 1.04. These results suggest that the predesign method can accurately predict the linearized slope m L and seems insensitive to the ξ project and h step combination.

Focus location
The other output from QTM is the estimate of the current focus location Z focus . Figure 13 shows the output of the tradespace analysis for Z focus and Fig. 16(b) the localized gradient of the surface. Unlike linearized slope, the Z focus location changes very little for the variable ranges except for the lower right-hand corner, i.e., low ξ project and high h step . For the gradient, ξ project very strongly affects the focus position with a mean gradient angle of −76.9 deg. Furthermore, most of the vectors are nearly 90 deg, meaning that ξ project is essentially the only effect on Z focus . This is likely caused by the projector's MTF having different responses for different ξ project , as seen in Fig. 5.
The dynamic range of the Z focus values in Fig. 13(a) is 20.9 μm. While only 10.4% of the camera's depth of field, that range may be too large for SLI systems that require micron-focus precision. Looking back to the through-focus curves in Fig. 5, we find that focus positions for ξ project ¼ 13.7 and 3.4 cy/mm are not equal and in fact are separated by 27.0 μm. Thus, the range of Z focus in Fig. 13(a) test system is due to the test system's frequency-dependent focus position and not an error within the predesign method.
As in Sec. 4.3.1, Z focus was measured for the seven combinations using the same experimental procedure. Z focus was determined with Eq. (2) with the results shown graphically in Fig. 14 and numerically in Table 5. Figure 14 displays a noticeable correlation between experimental and simulated focus locations, with the regressed slope at 1.05. However, the correlation is not as strong as m L with R 2 ¼ 0.55 likely due to experimental noise. The predicted accuracy of Z focus is very high with an RMS difference of 1.6 μm over the seven combinations, thereby suggesting that the predesign method can accurately estimate the current focus location.

Alignment Design Recommendation
The presented results suggest that the alignment process can be simulated. Thus, we can now predesign the QTM variables using the results in Figs. 12 and 14. For the tested SLI system,  maximum adjustment sensitivity is desired more than the usable Z range due to the relatively small lateral resolutions of 5.0μm; thus, h step ¼ 1000 μm and ξ project ¼ 13.7 cy∕mm.

Conclusions
The predesign of a linearized projector alignment method has been realized using simulation. The presented predesign method uses the native optical design file to determine the projector alignment variables so that a user may quickly move from design to a functioning alignment system. The method was compared with experimental results over nine different combinations of projected fringe frequencies and quad target step heights. The simulation method predicted both experimental outputs of the QTM accurately with the linearized slope RMS difference of 0.02 mm −1 and the current focus position to 1.6 μm RMS.