Laser range-gated imaging can penetrate a strong scattering medium and capture the image of an object at a certain distance by applying active laser illumination, changing the gate delay time, and varying the gate width. It has been widely used in areas such as military reconnaissance, monitoring, underwater detection, and search and rescue. For cases when the line of sight is blocked by corners of walls or other building obstructions within the environment of city streets and buildings (as shown in Fig. 1), traditional imaging techniques and equipment are unable to achieve imaging beyond the direct field of view. Non-line-of-sight imaging can bypass the corners of walls or other obstructions to achieve imaging beyond the direct field of view, which can be used to reduce casualties in modern urban street fighting, and to increase battlefield awareness and operational efficiency.
In 2006 and 2007, researchers at the Research Institute for Optronics and Pattern Recognition of Germany and Swedish Defence Research Agency (FOI)1,2 of Sweden carried out joint research on short-wavelength infrared laser range-gated imaging for ground applications in monostatic and bistatic configurations. A number of test targets (test panels, vehicles, and soldiers with weapons) were included in the tests, and windows in buildings and cars, traffic signs, and vehicle surfaces were utilized as the intermediary reflective surfaces. In 2011, the FOI group3 proposed the loss factor model through further study of the effect of the characteristics (including the shape and the reflection characteristics) of the intermediary reflective surface on laser illuminance attenuation and measured the loss factors with different intermediary reflective surfaces. Though their work was insightful, their model is not effective for describing the imaging process. Since 2007, the Media Lab at the Massachusetts Institute of Technology (MIT)4 has been studying transient imaging based on femtosecond laser range-gated imaging technology and proposed an algorithm for extracting object information from multireflection, realizing a means of seeing around the corner. In 2012, researchers at MIT5 realized a three-dimensional reconstruction of hidden targets utilizing diffuse reflections of ultrashort laser pulses using an intermediary reflective surface. Since 2012, researchers at the Beijing Institute of Technology (BIT)6,7 have been studying non-line-of-sight imaging. The BIT group uses a nanosecond laser range-gated imaging system to realize non-line-of-sight imaging of an USAF 1951 standard target at different distances with glass and ceramic tile as the respective intermediary reflective surfaces. These existing researches show the potential application prospects of non-line-of-sight imaging in areas such as urban warfare, counter-terrorism, and disaster relief.
Since non-line-of-sight imaging based on laser range-gated imaging is affected not only by the parameters of the laser range-gated imaging system but also by the reflection characteristics of the intermediary reflective surface and the scene, construction of an optical imaging model is important to the development and application of non-line-of-sight imaging. This paper describes the construction of an image contrast model of non-line-of-sight imaging from the viewpoint of a two-dimensional illumination distribution on the photosensitive surface, in order to analyze the performance of a non-line-of-sight imaging system utilizing a laser range-gated imaging system and to provide a theoretical basis for actual non-line-of-sight imaging system design.
Principle of Non-Line-of-Sight Imaging
The key to non-line-of-sight imaging is to make it possible to see a street-corner target or an indoor target by utilizing an intermediary reflective surface, as illustrated in Fig. 2. A is the observer, B is the target under observation, and C is the intermediary reflective surface. B is outside the range of objects that can be directly imaged by A, so that C is utilized to realize non-line-of-sight imaging of target B.
A laser range-gated imaging system, consisting of a nanosecond laser and an intensified charged coupled device (ICCD) imaging system, is located at location A. The ICCD imaging system is focused on location B. The beam from the nanosecond laser from location A experiences first-order reflections (consisting of first-order specular reflection and first-order diffuse reflections, where the first-order diffuse reflection contains first-order backscattering and first-order forward-scattering) at the intermediary reflective surface C. The first-order specular reflection and first-order forward-scattering illuminate target B and then are reflected at target B in a Lambertian fashion (the second-order reflection). First-order backscattering is received by the ICCD imaging system, forming a defocused image and reducing the image contrast of target B. The second-order Lambertian reflection strikes the intermediary reflective surface C and experiences third-order reflections (consisting of third-order specular reflection, third-order backscattering, and third-order forward-scattering) at location C. Third-order specular reflection is received by the ICCD imaging system, forming a focused image of target B. Like the first-order backscattering, the third-order forward-scattering is also received by the ICCD imaging system, reducing the contrast of the image of target B.
For certain intermediary reflective surfaces (like ceramic tile), the effective target signal illuminance (third-order specular reflection) may be overwhelmed by the first-order backscattering and third-order forward-scattering, resulting in the target being undetectable. The first-order backscattering is apparently larger than the third-order forward-scattering, so the first-order backscattering is the dominant factor reducing the contrast of the image of target B. Thus, a better non-line-of-sight image of target B can be obtained by filtering out the first-order backscattering, which can be realized by laser range-gated imaging. Meanwhile, an image intensifying technique is also needed because the effective target signal illuminance (third-order specular reflection) is typically weak. The intermediary reflective surface is also a very important factor in non-line-of-sight imaging, which can be generalized to include a full specular reflector like glass (1), a partial specular reflector like ceramic tile (2), and an ideal diffuse surface (3). The study in this paper is mainly carried out on intermediary reflective surface types 1 and 2.
Construction of an optical imaging model of the non-line-of-sight imaging process, which is based on the parameters of the intermediary reflective surface, the parameters of the laser range-gated imaging system, and the properties of the target, is vital for the theoretical analysis and system design of non-line-of-sight imaging based on laser range-gated imaging. This model assumes that the imaging will occur at night or in low-light conditions to reduce the impact of environmental illuminance.
Image Contrast Model of Non-Line-of-Sight Imaging
As shown in Fig. 3, the laser beam is projected onto an intermediary reflective surface forming the light spot and experiences first-order reflections; then the first-order reflections are projected onto the target forming light spot . The distance from the laser to the geometric center of is . The distance from the geometric center of to the geometric center of is . The vector tracing the direction of the incident light from to makes an angle with (the vector normal to ) and an angle with (the vector normal to ). The optical axis of the laser and the optical axis of the ICCD are assumed to be coincident, and the divergence angle of the laser and the field view angle of the ICCD are also assumed to be coincident. In this section, the laser transmission characteristics (attenuation and distribution) in the non-line-of-sight imaging process will be analyzed.
Pulsed Laser and Range-Gated Imaging System1), where denotes the transmission power of the pulsed laser, is the propagation time, is the energy of a single pulse, is the pulse width, is the solid angle of the divergence of the laser beam, and is half divergence angle of the laser beam. The function is described by Eq. (3). The detector includes pixels, is the gate width, and is the strobe gating delay time.
Reflection Characteristics of the Intermediary Reflective Surface
The reflection characteristics (consisting of specular reflection characteristics and diffuse reflection characteristics) of the intermediary reflective surface are among the important factors that can influence the non-line-of-sight imaging results. Without loss of generality, the intermediary reflective surface is assumed to be free of absorption, so that its reflection characteristics can be described by the specular reflectivity and the diffuse reflectivity . is a constant, obeys Lambertian reflection, and . The reflectance distribution function is described by
Target Signal Illuminance and Disturbance Caused by the Intermediary Reflective Surface under CW-Laser Illumination
Illuminance on the target plane
The illuminance on the target plane is denoted as , which consists of two parts: is caused by the first-order specular reflection of the laser and is caused by the first-order forward-scattering. The illuminance on should be obtained first.
According to Eq. (1), the illuminance on is given by
First-order specular reflection
The area illuminated on by the laser first-order specularly reflected by can be written as
Therefore, can be calculated as
The target distance is equal to the sum of and . If is much larger than the square root of , and is illuminated by the vertically incident laser beam (), Eq. (8) can be further simplified.
To facilitate the analysis and calculations, Fig. 3 is expanded along the optical path. Assuming that the optical axis of the laser coincides with the optical axis of the ICCD imaging system, a Cartesian coordinate system is established with the geometric center of the lens as the Cartesian coordinate origin. The optical axis is along the axis, and the axis and the axis are, respectively, parallel to the detector array’s length and width directions, as is shown in Fig. 4. is the angle between the axis and the projection of on the plane, and points in the direction opposite to the direction of the axis. and are, respectively, projections of the laser beam on the intermediary reflective surface and on the target.
According to the geometric relationship, the projection equations for and on the plane are, respectively, given by
and are random points, respectively, on and , and is a microsurface element around (as shown in Fig. 5).11), where is the angle between vectors and and is the angle between vectors and .
Target signal illuminance on the photosensitive surface17), where is the luminance on ( is the reflectance of the target or background, for or ), denotes the F-number, is the aperture diameter, and is the spectral transmittance of the objective lens.
Disturbance illuminance on the photosensitive surface caused by scattering of the intermediary reflective surface
The disturbance illuminance on the photosensitive surface caused by scattering of the intermediary reflective surface consists of two parts: (third-order forward-scattering) and (first-order backscattering). Since the intermediary reflective surface is not on the target plane, each point on corresponds to a diffuse plaque on the photosensitive surface, and the diffuse plaque can be expressed by a point spread function. A Cartesian coordinate system is established with the geometric center of the lens as a Cartesian coordinate origin. is the intersection of the intermediary reflective surface and the axis, and ’ is the image point of , as shown in Fig. 6.
According to geometrical optics, the center of the diffuse plaque for point on is given by
The radius of the diffuse plaque can be written as
The point spread function of the diffuse plaque for point can be expressed as
First-order backscattering illuminance on the photosensitive surface
According to Eq. (21), the first-order backscattering can be expressed as
is the solid angle defined by the rays emanating from point to the aperture defined by the lens. In fact, since the distances between the camera and every point on are not the same, the value of for each of the points is not the same. As the laser divergence angle is very small, an approximate treatment for is made here.
Third-order forward-scattering illuminance on the photosensitive surface
In Fig. 5, is a microsurface element around . is the illuminance on caused by . The illuminance is inversely proportional to the square of the distance as expressed in Eq. (25), where is the angle between vectors and and is the angle between vectors and . Then we obtain
Since can be calculated as
Image Contrast Model of Non-Line-of-Sight Imaging
Non-line-of-sight imaging can be achieved with laser range-gated imaging with a pulsed laser; therefore it is necessary to make a distinction between third-order reflection and first-order backscattering.
Illuminance on the photosensitive surface under pulsed laser illumination
Under pulsed laser illumination, the illuminance on the photosensitive surface caused by a single pulse is a function of time .
Signal and noise under pulsed laser illumination
Considering the time sequence of the laser range-gated imaging system and the discrete nature of the detector unit, the illuminance on the detector unit caused by the target and background within a pulse period can be calculated as
The illuminance on the detector unit caused by scattering of the intermediary reflective surface can be expressed, respectively, as
We simplify the imaging system noise and environmental noise by combining them into an equivalent incident illuminance , which is used as shown in the following equation:
The total disturbance of the non-line-of-sight imaging system can be expressed as
Apparent contrast of the image on a photosensitive surface
The apparent contrast of the image on a photosensitive surface can be described by
Analysis and discussion
In addition to systematic noise, the scattered disturbance illuminance from the intermediary reflective surface is the main factor in image quality degradation.
1. For photoelectric imaging systems based on human eyes, detection and identification of the extended source target depend on the target contour brought out by the apparent contrast of the image. Although people simply believe that 3 to 5% is the contrast threshold for target detection with human eyes, the more effective approaches are based on minimum resolvable contrast (MRC)8,9 or target task performance (TTP).10 Enhancing the apparent contrast of the image is conducive to improving the performance of non-line-of-sight imaging system.
2. It is impossible to separate from because they are synchronized with each other. However, may be reduced or eliminated by time gating, as shown in Fig. 7.
• : is separable from [as in Fig. 7(a)]. may be filtered out completely without reducing the laser illuminance efficiency, and thus, the highest contrast image is obtained.
• : When the gate opens upon the return of the target signal, part of the first-order backscattering is accepted by the detector, and the image contrast is decreased to some extent. To this end, the image contrast can be improved by adjusting [as in Fig. 7(b)].
can be totally filtered out by setting the strobe gating delay time to . An optimum contrast image can be obtained if the gate width is simultaneously reduced to . However, utilization rate of the laser illuminance decreases with increase of . The laser range-gated imaging system cannot receive the target signal illuminance with , and the efficiency decreases to 0. Since the laser range-gated imaging itself is low-signal-to-noise ratio (SNR) imaging, sacrificing the laser illuminance efficiency significantly reduces the SNR of the image. Therefore, practical application of non-line-of-sight imaging with laser range-gated imaging requires careful balance of the overall system.
Imaging Characteristic Analysis of Non-Line-of-Sight Imaging
Based on the image contrast model of non-line-of-sight imaging constructed above, this section analyzes the non-line-of-sight imaging results with different intermediary reflective surfaces. Typical parameters of non-line-of-sight imaging with a laser range-gated imaging system are shown in Table 1. The imaging depth of field is 36 m with a laser pulse width of 120 ns and a gate width of 120 ns; 40 laser pulses are received by the photodetector within a frame when the imaging frame rate is 25 fps and the laser pulse frequency is 1000 Hz, so that a laser illuminance equivalent to 4.8 μJ is received by the photodetector per frame.
Parameters of non-line-of-sight imaging with the laser range-gated imaging system.
|ρt (target reflectivity)||1|
|ρb (background reflectivity)||0|
|Φ (energy of a single pulse in μJ)||0.12|
|tp (laser pulse width in ns)||120|
|Laser pulse frequency in Hz||1,000|
|θ (divergence angle of the laser beam)||2 deg|
|D0 (diameter of objective lens in mm)||20|
|f’ (focal length of objective lens in mm)||80|
|τ0 (lens transmittance)||0.9|
|Size of photodetector (mm2)||14×10.5|
|tg (gate width in ns)||120|
|Image frame rate (Hz)||25|
Impacts of Reflection Characteristics of the Intermediary Reflective Surface on Non-Line-of-Sight Imaging
The impacts of the reflection characteristics of the intermediary reflective surface on non-line-of-sight imaging are considered with , , and . Since ·, the target signal illuminance and first-order backscattering are separable.
Figure 8 shows the curves of the target signal illuminance , the first-order backscattering , and the third-order forward-scattering for different values of the diffuse reflectivity of the intermediary reflective surface with a CW laser. Since and are unevenly distributed over the imaging field, the and in Fig. 8 are the respective means of and in the center area of the imaging field. The curves in Fig. 8 indicate the following conclusions:
1. Under a CW laser, the target signal illuminance monotonically decreases with , and first-order backscattering monotonically increases with . is larger than only when is (i.e., diffuse reflection takes up only 5% of the total reflection illuminance, and the specular reflection illuminance dominates), that is to say that non-line-of-sight imaging can be realized with only a CW laser when the intermediary reflective surface has very strong specular reflection characteristics. This conclusion indicates that practical realization of non-line-of-sight imaging must use a pulsed laser range-gated imaging mode in order to filter out . In the instance indicated in Fig. 8, can be totally filtered out without reducing the laser illuminance efficiency.
2. The target signal illuminance spans more than three orders of magnitude from for a completely specular reflection () to for a nearly completely diffuse reflection (). is several orders of magnitude smaller than , and is two orders of magnitude smaller than even when is equal to 0.99. Figure 9 shows that the distribution of on the photodetector for , , , , and is within the same order of magnitude for (the tilt angle of the intermediary reflective surface) equals 20 or 60 deg. The image quality is severely disturbed by quantum noise when the target signal illuminance is too low, so is selected as an effective non-line-of-sight imaging condition. with is one order of magnitude smaller than with . is always two orders of magnitude smaller than when changes from 0 to 0.7. In view of the above discussion, is often negligible.
Impacts of the Location of the Intermediary Reflective Surface and the Gate Width on Non-Line-of-Sight Imaging
The parameter is chosen to be equal to to fully use the laser illuminance and to reduce the noise for a typical laser range-gated imaging system, in which case the imaging depth of field is ·. The gate opens when the echo signal reaches the detector, so that the gate delay time equals . The following discussion is based on the assumption that is equal to 0.
Figure 10 shows a plot of the image contrast versus distance with and . The plot shows that first decreases with and then increases with to a certain constant value at when is totally separated from and that finally is nearly fixed with increasing . Figure 11 shows a plot of the image contrast versus distance with and . The plot shows that increases with and that obtains its maximum value when is totally filtered out at .
This paper describes the construction of an image contrast model of non-line-of-sight imaging with a laser range-gated imaging system on the basis of the modeling of an intermediary reflective surface. The image contrast model analyzes three factors affecting the image contrast of non-line-of-sight imaging with different intermediary reflective surfaces, where the three factors are target signal illuminance, first-order backscattering, and third-order forward-scattering. A non-line-of-sight imaging experiment simulation was carried out with the parameters of an actual laser range-gated imaging system. The simulation results show that (1) the first-order backscattering is the dominant disturbance to non-line-of-sight images and can be effectively filtered out by laser range-gated imaging and (2) the effect of the third-order forward-scattering on non-line-of-sight images is often negligible since the third-order forward-scattering is insignificant compared to the target signal illuminance in most cases. The image contrast model of non-line-of-sight imaging with a laser range-gated imaging system gives insights into the theoretical analysis and system design of non-line-of-sight imaging.
The simulation results also show that the reflection characteristics of the intermediary reflective surface have a significant effect on non-line-of-sight imaging. The bidirectional reflectance distribution function (BRDF) is a more comprehensive description of surface reflection characteristics; thus, BRDF modeling of typical construction material surfaces is a direction for further research. Meanwhile, since non-line-of-sight imaging with laser range-gated imaging system is usually low-SNR imaging, estimating the range of non-line-of-sight imaging using MRC or TTP is another direction for further research.
This work was supported by the Specialized Research Fund for the Doctoral Program of Higher Education (No. 20111101110013) and the Foundation of the Science and Technology on Low-Light-Level Night Vision Laboratory (No. J20110504). The authors would like to thank all the reviewers for their helpful comments and constructive suggestions. They would also like to thank Mr. Ye Peng and Dr. Xuan Hongwen for their assistance with the programming.
Kaida Xu received his BS from Mechanical Engineering College, China, in 2001. He is currently pursuing his PhD in optical engineering at Beijing Institute of Technology, China. His technical interests include low-level-light and infrared imaging technology.
Weiqi Jin received his PhD in optical engineering from Beijing Institute of Technology of China in 1990. He is currently a professor with the Department of Optical Engineering in the School of Optoelectronics at the Beijing Institute of Technology. His major research interests include low-level-light imaging, thermal imaging, and image processing. He is a fellow of SPIE.
Shenyou Zhao received his BS in control technology and instrument from Changchun University of Science and Technology in 2011. He is currently pursuing his master’s degree in optical engineering at Beijing Institute of Technology, China. His main technical interest is optical imaging technique.
Jing Liu received her BS in optical engineering from Beijing Institute of Technology of China in 2009. She studied augmented reality in Universidad Politecnica de Madrid of Spain from March to September 2009. She is currently pursuing her PhD in optical engineering at Beijing Institute of Technology of China. Her technical interests include underwater imaging, laser range-gated imaging, polarization imaging, and imaging process.
Hui Guo received her PhD in optical engineering from Beijing Institute of Technology in 2005. She is currently a doctoral supervisor in the National Key Laboratory of Science and Technology on Low Light Level Night Vision, Xi’an, China. Her major research direction is low-light-level night vision technology.
Su Qiu received his PhD from School of Optoelectronics at Beijing Institute of Technology in 2013. He is currently an assistant professor in School of Optoelectronics at Beijing Institute of Technology. His major research interests include digital image processing, low-level-light and infrared imaging technology.