Demonstration on range imaging of 256×256 pixels and 30 frames per second using short wavelength infrared pulsed time-of-flight laser sensor with linear array receiver

Abstract. We demonstrated the range imaging with high resolution of 256×256  pixels and high frame rate of 30 frames per second (fps) using a short wavelength infrared pulsed time-of-flight laser sensor, which is suitable for long range imaging. We additionally demonstrated the long range imaging of more than 1 km and wide field of view imaging of 12  deg× 4  deg, 768×256  pixels, and 10 fps. For these demonstrations, we developed the linear array devices of the aluminum indium arsenide avalanche photodiode array and silicon germanium bipolar complementary metal oxide semiconductor read-out integrated circuit array. We also deployed the flattop beam illumination optics with the beam division and recombination method and realized efficient line shape illumination.


Introduction
The time-of-flight (TOF) imaging laser sensor is an attractive tool for many applications, including robotics, terrain visualization, augmented vision, reconnaissance, pedestal detection, and so on. For the laser TOF measurement, there are basically two methods, pulsed and continuous wave modulation. It is generally said that the former is suitable for longdistance ranging, and the latter is better for the precise ranging in short-distance ranging. 1 Therefore, for the applications that require an imaging range of more than a few tens of meters, the pulsed method is considered to be better. Lamberts 2 first presented the pulsed TOF imaging laser sensor with a 1.06 μm Nd:YAG laser and a two-dimensional (2-D) scanner. The laser sensor with a 2-D scanner is widely used now especially for the survey application, but it is not useful for real-time imaging since it measures the TOF of only one point on the scene and takes a substantial amount of time to scan whole imaging areas. This is conceptually true for the cases of long range imaging that need a large scanning mirror. For example, we reported the imaging results using this type of laser sensor in the past literature, 3 and the frame rate was just 2 Hz for the imaging of 128 × 128 pixels.
The TOF laser sensor with an array (2-D or linear) receiver is a solution for the real-time imaging, since this can measure the TOF of multiple points on the object instantly. Stettner and Bailey 4 shows this type of TOF laser sensor with a 2-D array receiver, using a 1.06 μm Nd:YAG laser and silicon avalanche photodiode (APD) array. This concept has been deployed by many groups using the wavelength region of 0.53, 5 1.06, 6 and 1.57 μm. [7][8][9] The 2-D array receiver can realize instant range imaging by flashing the scene using one transmitted pulse without any scanning. However, there are the following issues in 2-D array type. First, a very high transmitted pulse energy is necessary to get enough receiving power for each element. For example, in the case of 256 × 256 pixels array, the required pulse energy is 65,536 times larger compared with the one in the case of a single pixel [for the same measurable range, in cases of the same transmitted average power, same receiving aperture size, same receiving sensitivity, and same field of view (FOV)]. Second, the yield rate of the array devices tends to be low because of its too many elements in the 2-D array device. Third, the performance of the read-out integrated circuit (ROIC) is restricted because the size of each pixel must be small (e.g., 100 μm × 100 μm 10 ). Additionally, the scanning is consequently needed even for the 2-D array type in many cases of wide FOV since the FOV of the 2-D array receiver is limited by the size of the 2-D array and the F number of the receiving optics.
On the other hand, the TOF laser sensor with a linear array receiver has merits on these issues. The required pulse energy can be lower compared with the one for the 2-D array type. For example, in the case of 256 × 256 pixels imaging, the 256 pixels linear array is used, and the required pulse energy can be 1∕256 compared with the one for the 2-D array case. The required average transmitted power can be the same if a high repetition rate laser is used for the linear array type. The efficiency of the laser transmitter is generally high for the low pulse energy and high repetition rate laser. It is much easier for the linear array type to get the array receiver of which all pixels in the array work well. The performance of ROIC can be better for the linear array type since the area of ROIC can be larger. This larger area enables the installation of larger numbers of components (transistors, capacitors, and so on) in a chip to realize the better noise characteristic and larger number of functions. Further, the sensor FOV can be flexible (in many cases, wider) easily by changing the scanning angle width. Eberle et al. 11 reported this type of laser sensor with 384 pixels linear array mercury cadmium telluride (MCT)-based APD, and showed preliminary imaging results at the distance of 270 m using the wavelength of 1.57 μm. However, the MCT-APD does not work well at room temperature although its specification is very attractive for the high receiving gain and multifunctionality (i.e., both working for active and passive imaging). The operating temperature of the MCT-APD is not denoted in Ref. 11, but it is reasonable to consider that the temperature is around 70 K by referring to another laser sensor using MCT-APD. 9 In addition, the maximum read-out rate is 4 kHz (corresponding to 1.54 MHz∕pixel), and this restricts the sensor frame rate to 15 Hz (below the video rate) for 384 × 256 pixels imaging. The detail of the ROIC and the reason for this restriction cannot be known in the reference. Further, the transmitting optics was designed to form the flattop-like illumination only at the far-field and is not suitable for the flattop illumination for wide distance range which is an important function for imaging laser sensors.
To overcome these issues, we have developed (i) 256 pixels linear array APD with aluminum indium arsenide for the multiplication layer (AlInAs-APD) for short wavelength infrared (SWIR) band which works at room temperature, (ii) silicon germanium bipolar complementary metal oxide semiconductor (SiGe-BiCMOS) ROIC array with the read-out rate of 2.56 MHz∕pixel. The AlInAs-APD also has the feature in its low excess noise factor compared with the APD with indium phosphide multiplication layer (InP-APD), which is used in the past laser sensor. The ROIC is an analogue type and has features in multifunctionality in addition to the high read-out rate. We have also deployed the flattop beam illumination optics with the beam division and recombination method 12 and realized efficient line shape illumination which is necessary for the linear array type of laser sensor. The concept of our development is shown in Fig. 1. By combining these developments, we demonstrate the range imaging with a high frame rate of 30 frames per second (fps) and high resolution of 256 × 256 pixels. We additionally      demonstrate the long range imaging of more than 1 km. To the best of our knowledge, the above-mentioned imaging specification of our sensor is the highest for the range imaging pulsed laser sensor which is suitable for long range imaging. For example, the number of pixels in Refs. 7  We developed the APD array by utilizing our guardring-free planer AlInAs-APD technology. 13 The conceptual design, theoretical performance, and its comparison with the evaluated results are shown in the past literature. 13 To our knowledge, there has been no literature on laser sensors using AlInAs-APD. This is a linear mode APD and can be operated well in the required temperature of 300 K. In addition, this has a feature of a low excess noise factor. In Ref. 13, it is shown that the excess noise factor of this APD is 2.9 at the multiplication factor of 10. The InP-APD is widely used in the laser sensors (e.g., in Ref. 7). As denoted in Ref. 13, the excess noise factor of InP-APD is larger, and the value is 5.5 for the same multiplication condition. This means that the multiplication factor can be set at higher value for AlInAs-APD than the value for the other one under the same noise level. This is an advantage in using AlInAs-APD for laser sensors, and can be achieved with the careful tuning of the bias voltage. Figure 2 shows the structure of the developed 256 pixels AlInAs-APD array. All layers are grown on a ntype InP substrate by the molecular beam epitaxy. An AlInAs multiplication layer, a p-field control layer, an InGaAs absorbing layer, and a p-window layer are grown on n-type distributed Bragg reflector (DBR). The top surface is coated with a passivating film, which acts as antireflection coating. The dimension of the receiving area, which is formed on the top surface, is 50 μm × 35 μm. The element pitch is 50 μm.
The current-voltage characteristic with a multiplication curve of APD is shown in Fig. 3. The responsivity dependence on wavelength is shown in Fig. 4. The photocurrent was measured by inputting the light into a pixel with a power of 34 μA and changing the bias voltage. The multiplication factor was obtained using the measured photocurrent and responsivity of the APD. The responsivity can be known from the flat photocurrent region in Fig. 3, by assuming that the photocurrent of this region corresponds to the  multiplication factor of 1. In Fig. 3, it is shown that the breakdown voltage V br at a reverse current of 10 μA is about 40 V. The dark current Id at 90% of V br is as low as about 30 nA at room temperature. The maximum multiplication factor is over 100 and the factor at 90% of V br is about 10. The maximum photocurrent for the linear input-output operation is 0.1 mA at the multiplication factor of 10 and this is the index of the dynamic range. In Fig. 4, the responsivity at a wavelength of 1.55-μm band is 1.07 A∕W, which corresponds to a quantum efficiency of 85%. Although the quantum efficiency is kept within the SWIR band (from 1 to 1.7-μm band), the DBR enhances the responsivity at a wavelength of 1.55-μm band by 15%. The capacitance at 90% of V br is 0.3 pF at a frequency of 1 MHz. The −3 dB cut-off frequency is more than 1 GHz at a gain of 10.

Read-Out Integrated Circuit Array
The ROIC array is fabricated in 0.18 μm SiGe-BiCMOS process. Each element of the ROIC array includes the transimpedance amplifiers (TIAs), the peak intensity detector, and the TOF detector which is the time-to-amplitude converter    These values agree with the theoretical ones, which were designed by considering the transimpedance resistance and noise characteristics of the BiCMOS transistor. The transimpedance gain and the bandwidth were obtained by inputting the sinusoidal current signal of different frequencies to the TIA with known current level and measuring the output voltage. The equivalent input noise current density was obtained by the measured noise spectrum and the transimpedance gain. Figures 5, 6, and 7 show the configuration of the element of the array, the timing chart for the operation, and the photograph of the ROIC array chip, respectively. The chip size is 6.7 mm × 3 mm. A chip of ROIC array includes the multiplexers for read-out. The number of element of the array is 64. The read-out rate is 2.56 MHz∕pixel and this performance is proved in the imaging experiment shown later in Sec. 4.1. Owing to this read-out rate, our TOF laser sensor can realize the frame rate of more than 30 Hz (potentially, 39 Hz) for 256 × 256 pixels. The reason for this realization of such high read-out rate is that this ROIC obtains the range and intensity data by analog method without any digital processing. The operation of the ROIC is the following. At first, the ROIC is initialized by a reset signal. The gate for range measurement (GATE) signal turns on the peak detector and the negative feedback loop formed by an operational amplifier (Op-Amp), an analog MOS switch, and a capacitor. Then, after the negative feedback shifts to the stable state, the measurement starts by transmitted laser light and switching this GATE signal into high level. The maximum peak voltage can be detected while the GATE signal is high level. The output signal of the Op-Amp is used for generating sample and hold trigger (S/H TRG) with the comparator and the threshold voltage. During operating of the peak detector, the TAC operates in parallel. The voltage stored at the capacitor in TAC is charged by the current source after the GATE signal is at high level. At the moment of the detection of peak voltage, the S/H TRG switches to low level. The S/H circuit in the peak detector and the S/H circuit in TAC hold the peak   voltage and the ramp voltage, respectively. Since the ROIC is an analogue type, the timing resolution and jitter are not restricted by the ROIC itself and are restricted by the characteristic of the external control circuit to input the timing signal to this ROIC. For ideal signal detection, the threshold voltage (i.e., the negative input voltage) of the comparator should be slightly higher than the offset voltage of the Op-Amp (i.e., the positive input voltage of the comparator). In this case, the minimum signal detection level can be low. If the offset voltages are the same between the elements in the array, the ideal signal detection can be easily realized with a common threshold voltage for all elements. However, in actuality, the offset voltage has unavoidable dispersion between elements. Figure 8 shows this dispersion of the offset voltage. This offset voltage was measured by changing the setting of the threshold voltage and finding out the boundary condition for the peak detector to work. If the offset voltage is much higher than the threshold, S/H TRG is not output in the cases of small signals. This means that small signals cannot be detected. If the offset voltage is lower than the threshold, SH TRG continues to be output. In this case, the S/H circuit in the TAC does not hold the ramp voltage and consequently, the ranging function does not work. To resolve this issue, we have added the function of setting the optimum threshold voltage for S/H TRG. That voltage is configured by a digital-to-analogue converter (DAC). The DAC accesses the random access memory which is integrated in each ROIC element. Using this function, we can set the threshold slightly higher than the offset voltage of the Op-Amp for all elements and the ideal signal detection can be realized. The ROIC has another additional function in switching the measurement range width. That can be realized by controlling the current in TAC. The current is controlled by the external input of digital value as shown in Fig. 9. Figure 9(a) shows the designed measurement range width versus the digital value. Figure 9(b) shows the measured results of input time (corresponding to TOF) versus output voltage of the ramp, respectively. The ramp voltage characteristic was measured by changing the input timing of a pulse signal. The range width was obtained by measuring the time width from the start and end of the ramp. In Fig. 9(b), the two lines correspond to the ramp voltage in case of TAC ¼ 9 and 53, respectively. The range widths, which are obtained from Fig. 9(b), are 450 and 75 m (corresponding to 3000 and 500 ns) for TAC ¼ 9 and 53. These values agree with the designed ones shown in Fig. 9(a). In the developed ROIC, we can set the measurement range width depending on   the flexible user's request in time. If the ranging width is set to be short, the gradient of the time-to-amplitude in the TAC becomes steep and it contributes to the high precision ranging. On the other hand, when the ranging width is set to be long, it is suitable for long-range imaging. These additional functions (setting the optimum threshold voltage for S/H TRG and switching the measurement range width) are realized owing to the feature of the larger area of linear array concept which was described in Sec. 1.

Array Receiver Package
The linear array receiver consists of the APD array and the ROIC array assembled in one package. The photograph of the package is shown in Fig. 10. The four ROIC chips (each has 64 elements) are located at both sides of the 256 pixels linear APD array. The ROICs and APD are wired to each other. The package size is 42 mm × 42 mm.

Transmitting Optics
In this sensor, it is important to illuminate the transmitted laser light uniformly on a scene along with the line shape FOV. For this purpose, we have developed also the flattop beam illumination optics with the beam division and recombination method 12 and realized efficient line shape illumination which is necessary for the linear array type of laser sensor. Figure 11 shows the schematic of this optics. Note  that this figure shows one-dimensional illumination and does not show the discussion about the direction perpendicular to the sheet. In this optics, a Gaussian beam is diverged by a concave lens. The diverged beam is divided and refracted by a prism. After passing through the prism, the divided Gaussian beams are crossed and overlapped to each other.
As the results of this crossing and overlapping, intensity distribution within a combined beam becomes uniform. The feature of this optics is in keeping the flattop illumination for wide distance range. The beam shape for the perpendicular direction is adjusted to make the divergence angle smaller than the FOV of an array element. Here, this is the collimated Gaussian shape. Figure 12 shows the evaluated result of the developed optics with the design of the divergence angle of about 4 deg. In Fig. 12, the optical coating was optimized for the wavelength of 1.06 μm and the result was obtained using this wavelength. It is shown that uniform illumination is realized. The evaluation was done at the short distance of 2 m but it is shown in the past literature 12 that the flat illumination can be kept for long range. Standard deviation of illuminated intensity within a designed illuminated area is about 4% except for the edge part, where distinct ripples appear because of a diffraction effect. This effect is not so large since the divided beams are the diverged ones.

Prototype Model
We developed the prototype using the key devices. The sensor configuration is shown in Fig. 13. The sensor has the array receiver, in which the APD array and the ROIC array are assembled in one package as shown in Sec. 2.3, the transmitting optics that can illuminate uniformly on an object, and the commercially available pulsed laser. We developed two prototypes of which one uses the 1.55-μm erbium-doped fiber laser and the other uses 1.06-μm diode-pumped Nd: YAG laser. The pulse energies of the former and the latter lasers are 0.2 and 1 mJ, respectively. Basically, the pulse energy is the only point that is different between the two prototypes, and the pulse widths and the pulse repetition frequencies are 8 ns and 10 kHz for both lasers. The purpose of using the former laser is the demonstration of the imaging using the eye-safe wavelength and the purpose for the latter is the demonstration of high performance (i.e., high resolution of 256 × 256 pixels, high frame rate of 30 fps, and long range) imaging using the higher energy laser. The aperture diameter of the receiving optics is 75 mm. The FOV of the array receiver is 4 deg for the vertical direction. These optical parameters are the same for both wavelengths, but the optical coatings are the customized ones for each wavelength. A horizontal dimensional scanning mirror is a galvano type. The personal computer (PC)-based signal processor includes  The information for horizontal direction can be known from the scanner angle at the timing of each pulse shot. The internal view of the main container of the prototype is shown in Fig. 14. The photograph is shown only for 1.55 μmtype, but the basic configuration is the same for both wavelengths (i.e., 1.55 and 1.06 μm).

Ranging Precision and Accuracy
Here, we show the evaluation result of the ranging precision and accuracy. The used prototype is the 1.55-μm wavelength type. The precision and accuracy are the same for both wavelength types, since the pulse width and the performance of the ROIC are the same. The range imaging result used in this evaluation is shown in Fig. 15. The setting of the range width of ROIC was roughly adjusted to the maximum range point in the imaging scene. This is same for all imaging results in the following section of this paper. The static scene was selected to evaluate the ranging precision and accuracy. This figure has 768 × 256 pixels with the FOV of 12 deg × 4 deg, but these are not important points in this evaluation. This figure also shows the proof of the read-out rate of 2.56 MHz∕pixel which was described in Sec. 2.2, since this figure shows that the read-out of 256 pixels is performed well in the pulse repetition rate of 10 kHz. We evaluated the ranging precision as the standard deviation of ranging results at the specific point (denoted in the figure) for successive 100 frames. The histogram of the ranging result is shown in Fig. 16. The standard deviation is 7 cm and this is the ranging precision of this system. This precision is dominated by the noise of the receiver. To obtain the range accuracy, we investigated the ranging results for the multiple pixels corresponding to a same building wall (denoted in this figure). These ranging results should be almost the same (<5 cm) by considering the measurement condition. For the evaluation of the accuracy, the ranging results of each pixel are averaged for successive 100 frames to reduce the instantaneous deviation. The histogram of the averaged result is denoted in Fig. 18. The standard deviation is 20 cm. Since the range of the system is calibrated at the mean value in the histogram of Fig. 17 using the ground truth, the standard deviation of 20 cm can be assumed as the ranging accuracy of the system. This accuracy is mainly caused by the calibration error concerning with the time-to-amplitude gradient of the TAC in ROC. These performances on the precision and accuracy are almost the same as the laser sensor shown in Ref. 14.

Real-Time Imaging
Here, we demonstrate imaging experiments using the prototype model. The wavelength used here is 1.06 μm. The online frame rate is 30 Hz for imaging of 256 × 256 pixels. The FOV is 4 deg × 4 deg, respectively. The condition of 10 kHz pulse repetition frequency and 256 × 256 pixels may seem to conflict with the frame rate of 30 Hz. The time region, corresponding to both ends of the horizontal scanning, is not utilized for the imaging and is consumed for the signal processing. This limits the current frame rate of 30 Hz, but this can be improved depending on the computing power of the signal processor. Figures 18, 19, and 20 show the real-time To evaluate the quality of the obtained range and 3-D image, we tested the simple method of an object detection algorithm with intensity and 3-D data. The algorithm flow is shown in Fig. 21. The process is explained as follows. At first, the range and intensity data are obtained. Using the range data and the information of the angle, we reconstruct the 3-D data. In the next step, data screening is performed. In this process, using the threshold of intensity data and the data in 3-D coordinate, the data that have low intensity or that have the 3-D data standing outside the scope of observation area are screened. Then, we extract the pixels having the height within the object size and intensity that are higher than the threshold we defined. Moreover, we perform the clustering of the pixels with the method of the densitybased spatial clustering of applications with noise 15 which is clustering within the constant distance in Euclidean space. Finally, the cluster data are filtered by the cuboid whose size is defined by the real object size. This detection algorithm needs the range precision to divide between the object and the other clutter objects since this clustering and filtering method is directly linked to the range precision. The detection results for which the object is the human are shown in Fig. 22. The rectangles on the intensity image that correspond to Fig. 18 indicate the result of the object detection. Although Figs. 18-20 were obtained in real time, Fig. 22 is the postprocessed result because of the current computation power of the PC. These figures show that the object can be detected even in the scene with some clutter. Figure 23 shows a long-range imaging of a landscape obtained by the same prototype as Sec. 4.2. Although the quality of the imaging is worse than that of Figs. 18 and 19 because of the lower signal-to-noise ratio, it is shown that the imaging was basically performed successfully even at a long range of more than 1 km. The horizontal streaks in Fig. 23 are caused by the dispersion of the minimum detectable signal level (note that each vertical pixel corresponds to each pixel of the linear array receiver). The ROIC, which was used in this imaging experiment, did not have the additional function of setting the optimum threshold voltage for S/H TRG which is denoted in Sec. 2.2. This is the reason of the above-mentioned dispersion of the minimum detectable signal level. Note that, for the all other imaging results shown in this paper, we used the ROIC with this additional function.

Long-Range Imaging
The validity of this imaging range is not studied in detail here, but basically, it can be confirmed to be reasonable by considering the laser radar equation 16 and the sensor parameters denoted in Secs. 2 and 3.

Wide Field of View Imaging
The wide FOV imaging is the advantage of the linear array type compared with the 2-D array type as described in Sec. 1. We demonstrate this advantage by changing the scanning width of the galvano scanner. The used wavelength is 1.55 μm for this experiment. Figures 24 and 25 show the intensity and range imaging result with 10 fps and 768 × 256 pixels. The FOV is 12 deg × 4 deg, respectively. It is shown that changing FOV worked well.

Conclusions
We demonstrated the range imaging with high frame rate of 30 fps and high resolution of 256 × 256 pixels using SWIRpulsed TOF laser sensor which is suitable for long range imaging. We additionally demonstrated the long range imaging of more than 1 km, and wide FOV imaging of 12 deg × 4 deg with 768 × 256 pixels and 10 fps. These performances are realized by the key devices that we have developed. The linear array AlInAs-APD array has a feature in the low excess noise and works well at room temperature. The advantage of the above-mentioned low excess noise has not been proved in this paper. However, this feature provides a future potential for better receiving performance, compared   with the case of InP-APD, by careful tuning of the bias voltage. The SiGe-BiCMOS ROIC array is an analogue type and offers real-time read-out of intensity and range information with a high read-out rate of 2.56 MHz∕pixel. This ROIC also has a function of optimizing the detection threshold for each pixel and tunable ranging width. We deployed the flattop beam illumination optics with the beam division and recombination method for the realization of efficient line shape illumination.