Optical three-dimensional (3-D) geometry measurements are state of the art when it comes to contactless quality control and maintenance of the shape of technical components that exclude tactile measurements due to filigree or internal structures. Optical inspection methods are also characterized by a fast and high-resolution 3-D inspection of complex geometries. And due to their noncontact principle, they can carry out measurements in places that would otherwise not be accessible due to harsh environmental conditions or specimens such as hot forged parts. However, there are currently no methods to estimate the reconstruction quality for the optical 3-D geometry measurements of hot objects. The mainly used geometric measurement standards cannot be used for the characterization of hot measurements since the calibrated geometrical values are not transferable to high temperatures. For the development of such a metric, we present the fundamentals of the concepts and algorithms for an estimation of the reconstruction quality are presented and evaluated using a two-dimensional simulation model. The generated findings were applied to the 3-D geometry measurement of a hot object in a laboratory environment. The results are compared with general state-of-the-art reconstruction quality metrics.
In this paper, we introduced a suitable metric to estimate the reconstruction accuracy of an optical three-dimensional (3-D) measurement system used under the influence of a refractive index gradient. Such conditions may be caused by the heat transfer from a hot measurement object into the ambient air, which leads to a deflection of the measurement light from its assumed linear path. This light deflection effect is an additional source of uncertainty when measuring the 3-D geometry of a hot-state object. The uncertainty estimation cannot be based on the comparison of the measured data with the calibrated geometry of a measurement standard, as is regular practice, since the calibrated features correspond to precisely the calibration at defined conditions, usually ambient pressure and temperature . Also, the geometry of any component at hot-state cannot be measured by a tactile coordinate measurement machine since it cannot be excluded that the elevated temperatures have damaging effects on the tip or may cause conditions for which the system is no longer calibrated.
Members of this research group have tried to estimate the added uncertainty by analyzing the acquired data by comparing it with a mathematical model1 and have conducted a multiphysics simulation2 to estimate the effect of the light deflection on a simulated 3-D geometry measurement. In addition, we have measured hot objects in different ambient pressure situations3 and from different angles4 to try to estimate metrics to evaluate the reconstruction quality. An indirect metric was found in the estimation of the inhomogeneous refractive index field induced by the heat transfer of the hot object into the ambient air using the background-oriented schlieren method.5,6 Nevertheless, a direct proof and estimation of the influence of the light deflection effect was not conducted.
Other groups have also developed 3-D measurement techniques for hot-state components, but they omitted a reconstruction quality estimation or based it on the predicted geometry of a heated component. The geometry prediction was based on a combination of the measured object temperature, the thermal expansion coefficient of the components material, and the measured geometry of the component at room temperature. Standard quality estimation methods for optical 3-D geometry measurement systems, e.g., the backprojection error from Hartley and Zisserman,7 were not applied. The results from the literature are recapitulated in the following to give an impression of the objectives of other researchers and the achieved uncertainties and methods to estimate those uncertainties.
Liu et al.8 used a line projection device combined with a two-camera stereo vision setup to calculate the diameter of a hot cylindrical forged part. The radiation from self-emission of the hot object was blocked by a lowpass filter. There was no hot reference object available for the experiment, wherefore the accuracy was not estimated. Their proposed solution was based on the calculation of the fundamental matrix from the corresponding pixel points7 and the matching of points to the epipolar constraint. Their second validation approach was based on a heating simulation and the expected expansion of the measurand. Here, their results match with an error of . Zhang et al.9 used a laser light section method for the geometry reconstruction of cylindrical and rectangular hot heavy forged parts. The main research topic was the investigation of movement of the measurement system and the width of the laser line projection. An accuracy estimation was not conducted. Zatočilová et al.10 developed a stereo vision system for rotationally symmetric forgings under high temperature. However, since they were not able to identify homologous points in both cameras, they opted to reconstruct the hull of the object by fitting two perperdicular planes in the images. They estimated that their possible accuracy was in the millimeter range.
Schöch et al.11,12 developed a holistic geometry measurement system for arbitrary component shapes based on multiple light section scanners and a linear motion platform. They estimated the accuracy of their system at room temperature to be around 0.15 mm for a diameter measurement of 300 mm. For validation under hot conditions, they measured low-expansion workpieces (lengths with a thermal expansion coefficient of ) at about 850°C. While the reconstructed object lengths were within the expected range, there was no explicit statement of the achieved accuracy or the influence of the light deflection effect by the inhomogeneous refractive index field. They additionally estimated the theoretical maximum influence of the light deflection effect on light-section measurements to be about for objects at a temperature of 100°C. Their calculation was based on the light deflection caused by a homogeneous refractive index field of a slightly different refractive index outside the sensor housing. They concluded that the light deflection effect can be neglected due to its diminishing importance compared to the size of the measurand of about 300 mm in diameter.
Du et al.13 used a two-dimensional (2-D) laser radar, comprising a single laser beam and two servo-motors, to measure the geometry of cylindrical hot heavy forged parts. The measurement error was estimated to about 2 mm for a cylindrical reference object of () at room temperature. For the same object at hot state, they estimated the error to be 5 mm for the diameter and 8 mm for the height. The single point error was not investigated, and the reference method was not stated. Zhang et al.14 combined a light section system with reflection models to reconstruct the edges of a hot component. This simplifies the evaluation of the acquired 3-D data to calculate the length of an object. The accuracy of the method was estimated by comparing the reconstructed the geometry of the cold object with a reference measurement; therefore, neither a single point accuracy nor a hot reference method was given. Hawryluk and Dworzak et al.15,16 used the combination of a light-section laser-scanner with a measuring arm, which is a six-axis serial arm reconstructing the 3-D position of its tip by measuring its joint angles. The system was used to reconstruct the geometry of the tools used in a hot-forming process. A wear analysis of the tools was conducted based on the acquired geometry. In our opinion, the wear thus described is not sufficiently examined in terms of its plausibility due to uncertainties caused by inhomogeneous ambient conditions or inherent uncertainties. Bračun et al.17 used a laser-scanner to reconstruct the geometry of a hot specimen. They focused on the compensation of different lighting conditions due to inhomogeneous emission coefficients on the object’s surface as well as the measurement accuracy. They identified scale chips on the object’s surface and the surface mircotopography as the most significant influences. Zhou et al.18 used a stereo vision system in combination with a line projector. The main research was focused on the line detection along measurement object edges in the stereoscopic image pairs. The detected lines were used to reconstruct the object’s geometry edges through epipolar geometry.7 Zhang et al.19 introduced a laser scanner-based shape reconstruction method for large ring forgings. The main aspects of their approach were given by data fusion of consecutive measurements from different sensors to the holistic geometry of the component. For the validation of their data, they ompared the measurement results to the components radius, which was measured by a caliper. The estimated errors were within 1.2 mm for a radius of about 450 mm.
Basics of Optical 3-D Geometry Measurement
Optical 3-D geometry measurement techniques are state of the art when it comes to contactless quality control for the shape of technical components, especially for free-form objects.20 The mapping of an arbitrary 3-D point onto a pixel location on the sensor of a camera is described by the perspective projection of the pinhole camera model, according to
The optical measurement of a 3-D geometry therefore needs to compensate for the loss of dimensions, e.g., through the use of two or more cameras observing the same scene. The reconstruction is then based on the triangulation of homologous 2-D points in multiple cameras. A successful reconstruction needs the satisfaction of Eq. (1) for each corresponding point; therefore, the identification of the system-inherit parameters [, for each camera ] and the detection of the homologous points are necessary.
A calibration routine using a known target is used to compute those camera parameters, e.g., the one developed by Zhang.21 The reconstruction itself is then based on the intersection of two lines from two cameras , resulting in a reconstructed 3-D point . Since these lines in 3-D space are usually skewed due to a leftover calibration error or numerical uncertainties, different algorithms were developed to reconstruct the 3-D point. A popular method is an optimization based on the epipolar geometry.7
However, homologous points may not be detectable on measurement objects without a structured surface. An active triangulation system, which incorporates an additional illumination unit, is used to counter this. One approach uses different projection images to add an artificial structure to the surface, e.g., by projecting a pseudorandom pattern to create homologous points. Such structured surface can then be reconstructed using the described triangulation technique with multiple cameras. A different approach uses a coded projection sequence to retrieve the corresponding projector pixel in a given camera pixel location , e.g., the multifrequency phase-shift method developed by Peng et al.22 The projector is modeled here as an inverse camera to then reconstruct 3-D-points from the given pixel values of camera and projector , i.e., , using the described triangulation techniques.
For the researched metric to estimate the reconstruction accuracy of an optical 3-D geometry measurement technique, in our case a fringe projection system, it is necessary to identify corresponding points in a setup comprising one projector and multiple cameras. To this means, Bräuer-Burchard et al.23 and Reich et al.24 investigated the additional knowledge gained from a system with more than one camera and a single projector . They developed an algorithm to change the basis of the calculation from the camera sensor to the virtual projector sensor . Using this method unlocks additional stereo pairs for triangulation by finding the corresponding camera pixels, i.e., not only reconstructing the 3-D points but also using . There are stereo pairs in such a system with optical devices. Therefore, there are reconstructed 3-D points for each projector pixel. To benefit from the fractional projector pixel locations retrieved by the phase-shift algorithm, they developed a method to arbitrarily scale the projector resolution. The main advantages of this method are the optimization of the point cloud density by changing the projector resolution in a virtual raster (VR).
First, a simplified model of the influence of refractive index fields on the reconstruction of 3-D points is described. The model is based on the compression of an inhomogeneous refractive index field to a medium containing two refractive indices with a discreet and sharp boundary layer. The assumptions are used to explain the background of the proposed method. Based on this, the presented method is brought into relation with the known backprojection error and the deviation of corresponding object points.
Suppose a ray of light, traveling through a medium with refractive index and a medium with , is subject to refraction at the boundary interface between and . The angle of refraction is calculated from the incident angle and the quotient for refractive indices , according toFig. 1). The view ray for the camera with center then connects the camera center and the first intersection of the ray of light with the interface layer , according to
For a 3-D geometry measurement system based on stereo triangulation, the reconstruction of an arbitrary point in a system with more than two cameras results in reconstructed points , each one from a combination of cameras to a stereo pair . For assumed homogeneous refractive index conditions, all reconstructed points are, theoretically, equal to the corresponding and observed point in space, according to
Estimation of the Reconstruction Quality for Multistereo-Pair Systems
The main requirement for all of the proposed methods is the unambiguity of the corresponding pixel locations. This is easily achievable for random-pattern or single-point triangulation systems due to the unambiguous nature of these systems. For fringe projection systems, this requirement can be achieved using Bräuer-Burchardt’s VR method (see Sec. 2), in which all camera pixels are calculated as a function of the projector pixel .
Different calculation methods for a quality estimation are shown by Hartley and Zisserman.7 The commonly used method is the backprojection, i.e., the recalculation of the pixel location from the reconstructed 3-D point [see Eq. (1)]. The Euclidean distance to the activated pixel used for the reconstruction is then called backprojection error, according to3.1 into account, the backprojection can also be based on the mean of correspondingly reconstructed points, i.e., on Fig. 2.
In this section, the setup for the experiments is shown. The proposed methods are tested as a 2-D simulation model under homogeneous refractive index conditions, i.e., , , . In addition, an optical 3-D fringe projection setup is used in a laboratory environment under homogeneous conditions, i.e., with a glass window, and under inhomogeneous conditions, i.e., measuring a hot object. In the simulation model, the light propagation is assumed to be in one plane as it simplifies the refraction and line–line intersection calculation.
To test the proposed approach, a simplified 2-D simulation is set up (see Fig. 3). For the model, a homogeneous and plane parallel refractive index field of thickness and refractive index is inserted into an otherwise also homogeneous propagation medium with a refractive index of . The path of light from a point in 2-D space to the center of a virtual camera is calculated using Snellius’ law (dashed lines) and a brute-force method. The mean of the leftover point-line distance from camera center to the refracted path of light is about . The vector from the camera center to the first intersection between is then used as directional vector (dashed lines) to calculate the virtual intersections
3-D Measurement Setup
The used optical 3-D fringe projection setup combines one green-LED projector (Wintech PRO 4500 based on TI DLP LightCrafter 4500) and four cameras (Allied Vision Prosilica GT 2050 and Prosilica GT 2300 with MeViS-C lens) to a multicamera fringe projection system [see Figs. 4(a) and 5(a)]. There are bandpass filters on the camera lenses used to block incoming radiation from the self-emission of the hot objects. The triangulation bases and triangulation angles are listed in Table 1.
Triangulation base and triangulation angle for all available stereo pairs in the fringe projection setup. The code for the stereo combination is taken from Fig. 4(a).
|Stereo pair combination||Triangulation base in mm||Triangulation angle in deg|
For an areal measurement, Peng et al.’s22 multifrequency phase-shift method is used to calculate the projector pixel in a given camera’s pixel by projecting a coded sinusoidal sequence onto a specimen. The measurement object with a pattern of the projected sequence is shown in Fig. 4(b). Bräuer’s23 VR method is then used to calculate the phasemaps , therefore having calculated corresponding pixels for each projector pixel .
The 3-D-points are reconstructed from each available view-ray pair using epipolar geometry.7 Camera–camera pairs on the same side of the projector ( and ) are not used here since the expected uncertainty is higher compared with the other pairs due to a smaller triangulation angle and triangulation base (see Table 1). The number of measured 3-D points for each projector pixel is shown in Fig. 5(b).
A series of experiments is conducted to compare and evaluate the proposed methods. In these experiments, the geometry of a cylinder of diameter is measured, which is placed in a prism holder with an anchor point (see Fig. 4(b)). The reconstruction quality is estimated for this cylinder under regular conditions and with the cylinder at the forging temperature of . To test the methods under the influence of a known refractive index field, a perspex disc with a thickness of and a refractive index of is placed between the measurement unit and the cold specimen (not shown). Each experiment consists of five consecutive measurements per measurement scenario, each taking about a span of 2 s. The experimental setup is kept constant between each set of measurements except for the measurement object, which is placed in an oven with for about 1 h. Each measurement contains the reconstruction of for each stereo pair . The proposed reconstruction quality metrics are then calculated from the identical measurement results, the per-pixel mean of those metrics are shown in the result maps (Figs. 7Fig. 10–9), and all measured points are evaluated for the histograms in Fig. 10.
The results for the 2-D simulation model and the 3-D fringe projection system are shown in this section. The expectations for the measurement results are based on the experience of this research group in regards to the geometry measurement of hot objects. In general, the reconstruction quality is expected to decrease when measuring a hot object through the heat-induced refractive index field compared with the measurement of the same cylinder at room temperature, as long as the cylinder is placed in a similar pose for both measurements. The exact same placement is not reached nor is the same diameter of the cylinder measured since both are influenced by the thermal expansion of the cylinder from to . The measurement with the inserted glass window should yield a lower quality reconstruction value than both other measurements. This is mainly due to Eq. (4), by which the amount of light deflection is described. Overall, it is not expected to reach a perfect reconstruction quality since there is an intrinsic leftover error from the (always) imperfect calibration routine.
Results for the 2-D Simulation Model
The points of investigation are tested in the 2-D simulation model (see Sec. 4.1 and Fig. 3). To this end, different points with and window thicknesses are compared over variations of the refractive index difference .
The results for a constant surrounding refractive index are displayed in Fig. 6(a). Here, the variances are nearly zero for and increase with increasing refractive index differences . The deviations increase equally for both and until a threshold of . Afterward, the deviation gradient stays constant for and decreases for until a saddle point at about .
For varying thicknesses , the deviations increase with increasing thickness . There are also larger interpoint variances for an increasing thickness based on the location of the backprojected point . These are detected by observing the thickness of the more transparent points in the background.
The deviations for a scenario with an ambient refractive index of (air at standard conditions) are shown in Fig. 6(b). Here, similar observations can be made compared to Fig. 6(a). However, the decrease of deviations when overcoming the threshold is not a part of the discussion of results. The relative interpoint differences are larger compared with those in Fig. 6(a), while being smaller on an absolute scale. The minimum deviation is at the expected place of .
The results from the laboratory experiment are split into the three proposed quality estimation methods since they vary in magnitude. The maps of the single camera backprojection error [from Eq. (9)] are shown in Fig. 7. The region of interest, from which the data are extracted, is shown in Fig. 5(b). The regular backprojection error is not homogeneously distributed and has a mean value of about 0.443 pixel for . There seem to be few differences compared with the quality estimation of the cylinder at with a mean deviation of [see Fig. 7(b)]. There are larger differences in the measurement with the inserted window in the light path of about 2.05 pixel [see Fig. 7(c)].
The maps of the backprojection error for each averaged corresponding object point and Eq. (12) are shown in Fig. 8, while the maps from the metric deviations of the corresponding 3-D points and Eq. (13) are shown in Fig. 9. In both cases, the differences from the measurements of the cold cylinder to the measurements of the warm cylinder [compare Figs. 8(a) to 8(b) and Figs. 9(a) to 9(b)] seem to be small but detectable, while the differences between deviations from the cold measurement to the measurement with the inserted window are larger.
The histograms of these results (see Fig. 8) are sorted by the used evaluation metric. Therefore, the comparable results from are shown in Fig. 10(a), and the results from and are shown in Figs. 10(b) and 10(c).
Discussion of Results
Considering the results of the 2-D simulation model in Sec. 5.1, the proposed quality metric as the deviations of the redundantly reconstructed points for meets the expectations. The single point deviation is highly dependent on the incident angle of each of the view rays for a camera , as is the parallel shift . However, the average of the reconstruction error for all points is strictly monotonously increasing with increasing nominal optical wavelength differences for . The relation between single point deviation and is recognizable in the thickness of the lines in Fig. 6, where each single-point reconstruction error is displayed in the background. The simulation model has therefore proven the validity of the proposed method for the detection of small refractive index differences in the reconstruction of points by triangulation.
In the case of the real experiment in Fig. 4, which includes a 3-D reconstruction of a hot specimen, it is demonstrated that the metrics introduced for quality estimation have a significant sensitivity for the detection of significant homogeneous refractive index influences [compare Figs. 7(a), 8(a), 9(a) to 7(c), 8(c), 9(c), respectively]. The differences are easily detectable in both the error maps , , as well as in the histograms in Fig. 10.
For the overall objective of detecting the influence of the heat induced refractive index gradient, the pixelwise differences between and on the respective estimation method seem to be insignificantly small. Even a statistical analysis of the acquired results yielded no comprehensible conclusion for [see Fig. 10(a)], while being subject to interpretation for and [see Figs. 10(b) and 10(c)]. In the histograms, the distribution of both error metrics seems to be (approximately) Gaussian, and there is a difference of for and for . The difference is both significant and robust considering the evaluated number of points of about . The relative small reconstruction quality difference is mainly due to the relative large error in the reference results, which are a consequence of an imperfect calibration process. Extrapolating from the 2-D results, the measurement with and yields and the measurement with and yields , i.e., a relative difference of . Comparing this with the relative differences in of and in of yields a much larger sensitivity of the model experiment compared with the measurements under laboratory conditions. This is mainly due to the additional influence of the calibration error in the workshop experiments.
The intent was to directly compare the displayed results as a difference map, e.g., as . This task proved to be impractical. The reasons for this effect are manifold.
The occurring and to be quantified light deflection moves the reconstructed points on the VR, so . In addition, the distribution of the reconstruction errors is inhomogeneous, even for the experiment in standard conditions [see, e.g., Fig. 7(a)]. Therefore, a small change in the object’s geometry or placement results in a change of the reconstruction error at that pixel position. These changes occur easily, mainly due to the different measurement conditions of the experiment at different states. Another reason is the surface of the object, which is constantly changing throughout heating and cooling periods as is its diameter. In addition, the position of the object cannot be assumed to be fixed, even though the cylindrical measurement object is placed in a prism holder with an anchor point.
Conclusion and Future Work
In this paper, a method for the estimation of the reconstruction quality of an optical 3-D geometry measurement system under inhomogeneous refractive index conditions was presented. The method is based on a multicamera single-projector fringe projection system without using additional geometric standards, e.g., a sphere or a cylinder with a calibrated diameter. The reconstruction metrics are based on redundantly measured 3-D points and the analysis of the geometric mean of these points.
The theoretical background for the proposed methods, as well as the results of a 2-D simulation model, was laid out. In the experiment, the influence of the inhomogeneous refractive index gradient was joined in a homogeneous refractive index field of varying thicknesses and refractive index gradients. The deviation of the reconstructed points was introduced as a reconstruction quality metric for the 2-D case, and its feasibility was verified by the results of the simulation model. Here, the mean deviation is proportional to the optical wavelength difference for small refractive index differences .
For the 3-D laboratory experiment, the feasibility of the proposed quality metrics was proven by the comparison of an uninfluenced measurement with a measurement in which a glass plate was inserted into the optical path. However, the comparison of a measurement of a cylinder at with the measurement at revealed no differences for the state-of-the-art quality metric of the single-camera backprojection error . The evaluation of the results for the quality metric based on redundantly reconstructed points showed a small but significant difference between the hot and cold measurements. Here, the metric deviation yields a slightly higher sensitivity to the influence of the refractive index gradient than the backprojection of the mean points .
Overall, the results from the proposed reconstruction quality metric yield detectable differences when measuring hot objects compared with cold objects. This enables the estimation of a relative reconstruction quality for measurements in which geometric measurement standards cannot be used. For these, it is necessary to establish a ground truth for each measurement scenario by measuring a similar object in cold state.
In the future, the proposed and presented reconstruction quality metrics will be used to assess different compensation methods for the light deflection effect induced by an inhomogeneous refractive index field around wrought-hot objects. The combination of the presented metric and a similar geometric measurement standard can be used to calculate an absolute quality metric in reference to said standard. Also, it is intended to reduce or restructure the intrinsic reconstruction error to enable an analysis of hot objects without the need for a comparison with a cold object of similar shape.
This article was funded by Deutsche Forschungsgemeinschaft (DFG); Collaborative Research Centre 1153 (CRC) Process Chain to Produce Hybrid High-performance Components Through Tailored Forming; Project number 252662854; Subproject C5 Multiscale Geometry Inspection of Joining Zones.
Lorenz Quentin is a research associate at the Institute for Measurement and Automatic Control of the Leibniz Universität Hannover (LUH). He received his diploma in mechanical engineering from the LUH in 2016. He is part of the Collaborative Research Centre 1153 (CRC) Process Chain to Produce Hybrid High-Performance Components Through Tailored Forming and researches mainly the optical 3-D geometry measurement of hot specimens using fringe projection.
Rüdiger Beermann is a research associate at the Institute of Measurement and Automatic Control at the LUH. He received his diploma in mechanical engineering from LUH in 2013, and his state examination as a teacher for math and metal technology for vocational schools in 2015. His current research interests include the development of fringe projection systems for high-temperature workpieces and thermo-optical simulations.
Markus Kästner is the head of the Production Metrology research group at the Institute of Measurement and Automatic Control at the LUH. He received his PhD in mechanical engineering in 2008 and his postdoctoral lecturing qualifications in 2016 from the LUH. His current research interests are optical metrology from macro- to nanoscale and optical simulations.
Eduard Reithmeier is a professor at the LUH and head of the Institute of Measurement and Automatic Control. He received his diplomas in mechanical engineering and in math in 1983 and 1985, respectively, and his doctorate degree in mechanical engineering from the Technische Universität München in 1989. His research focuses on system theory and control engineering.