Image contrast model of non-line-of-sight imaging based on laser range-gated imaging

Abstract. Laser range-gated imaging systems can obtain images of targets hidden around the corner, with an intermediary reflective surface with certain specular reflection characteristics. This imaging mode is called non-line-of-sight imaging. This paper describes a simulation of the target signal illuminance and disturbance radiation on the photosensitive surface of a non-line-of-sight imaging system based on modeling of an intermediary reflective surface. Meanwhile, an image contrast model of a non-line-of-sight imaging system is constructed. Simulation of the image contrast for a laser range-gated imaging system as a non-line-of-sight imaging equipment was carried out by analyzing the effects of varying the target signal illuminance and intermediary reflective surface reflection. Our simulation results show that the reflection characteristics of the intermediary reflective surface have a significant effect on the non-line-of-sight imaging. Although ordinary active laser imaging can realize non-line-of-sight imaging for an intermediary reflective surface with significant specular reflection characteristics, a laser range-gated imaging system is indispensable for non-line-of-sight imaging with commonly used intermediary reflective surfaces without significant specular reflection characteristics. The image contrast model of non-line-of-sight imaging constructed in this paper provides insights into the theoretical analysis and system design, as well as practical application of non-line-of-sight imaging.


Introduction
Laser range-gated imaging can penetrate a strong scattering medium and capture the image of an object at a certain distance by applying active laser illumination, changing the gate delay time, and varying the gate width. It has been widely used in areas such as military reconnaissance, monitoring, underwater detection, and search and rescue. For cases when the line of sight is blocked by corners of walls or other building obstructions within the environment of city streets and buildings (as shown in Fig. 1), traditional imaging techniques and equipment are unable to achieve imaging beyond the direct field of view. Non-line-of-sight imaging can bypass the corners of walls or other obstructions to achieve imaging beyond the direct field of view, which can be used to reduce casualties in modern urban street fighting, and to increase battlefield awareness and operational efficiency.
In 2006 and 2007, researchers at the Research Institute for Optronics and Pattern Recognition of Germany and Swedish Defence Research Agency (FOI) 1,2 of Sweden carried out joint research on short-wavelength infrared laser range-gated imaging for ground applications in monostatic and bistatic configurations. A number of test targets (test panels, vehicles, and soldiers with weapons) were included in the tests, and windows in buildings and cars, traffic signs, and vehicle surfaces were utilized as the intermediary reflective surfaces. In 2011, the FOI group 3 proposed the loss factor model through further study of the effect of the characteristics (including the shape and the reflection characteristics) of the intermediary reflective surface on laser illuminance attenuation and measured the loss factors with different intermediary reflective surfaces. Though their work was insightful, their model is not effective for describing the imaging process. Since 2007, the Media Lab at the Massachusetts Institute of Technology (MIT) 4 has been studying transient imaging based on femtosecond laser range-gated imaging technology and proposed an algorithm for extracting object information from multireflection, realizing a means of seeing around the corner. In 2012, researchers at MIT 5 realized a three-dimensional reconstruction of hidden targets utilizing diffuse reflections of ultrashort laser pulses using an intermediary reflective surface. Since 2012, researchers at the Beijing Institute of Technology (BIT) 6,7 have been studying non-line-of-sight imaging. The BIT group uses a nanosecond laser range-gated imaging system to realize non-line-of-sight imaging of an USAF 1951 standard target at different distances with glass and ceramic tile as the respective intermediary reflective surfaces. These existing researches show the potential application prospects of non-line-of-sight imaging in areas such as urban warfare, counter-terrorism, and disaster relief.
Since non-line-of-sight imaging based on laser rangegated imaging is affected not only by the parameters of the laser range-gated imaging system but also by the reflection characteristics of the intermediary reflective surface and the scene, construction of an optical imaging model is important to the development and application of non-line-of-sight imaging. This paper describes the construction of an image contrast model of non-line-of-sight imaging from the viewpoint of a two-dimensional illumination distribution on the photosensitive surface, in order to analyze the performance of a non-line-of-sight imaging system utilizing a laser rangegated imaging system and to provide a theoretical basis for actual non-line-of-sight imaging system design.

Principle of Non-Line-of-Sight Imaging
The key to non-line-of-sight imaging is to make it possible to see a street-corner target or an indoor target by utilizing an intermediary reflective surface, as illustrated in Fig. 2. A is the observer, B is the target under observation, and C is the intermediary reflective surface. B is outside the range of objects that can be directly imaged by A, so that C is utilized to realize non-line-of-sight imaging of target B.
A laser range-gated imaging system, consisting of a nanosecond laser and an intensified charged coupled device (ICCD) imaging system, is located at location A. The ICCD imaging system is focused on location B. The beam from the nanosecond laser from location A experiences first-order reflections (consisting of first-order specular reflection and first-order diffuse reflections, where the first-order diffuse reflection contains first-order backscattering and first-order forward-scattering) at the intermediary reflective surface C. The first-order specular reflection and first-order forwardscattering illuminate target B and then are reflected at target B in a Lambertian fashion (the second-order reflection). First-order backscattering is received by the ICCD imaging system, forming a defocused image and reducing the image contrast of target B. The second-order Lambertian reflection strikes the intermediary reflective surface C and experiences third-order reflections (consisting of third-order specular reflection, third-order backscattering, and thirdorder forward-scattering) at location C. Third-order specular reflection is received by the ICCD imaging system, forming a focused image of target B. Like the first-order backscattering, the third-order forward-scattering is also received by the ICCD imaging system, reducing the contrast of the image of target B.
For certain intermediary reflective surfaces (like ceramic tile), the effective target signal illuminance (third-order specular reflection) may be overwhelmed by the firstorder backscattering and third-order forward-scattering, resulting in the target being undetectable. The first-order backscattering is apparently larger than the third-order forward-scattering, so the first-order backscattering is the dominant factor reducing the contrast of the image of target B. Thus, a better non-line-of-sight image of target B can be obtained by filtering out the first-order backscattering, which can be realized by laser range-gated imaging. Meanwhile, an image intensifying technique is also needed because the effective target signal illuminance (third-order specular reflection) is typically weak. The intermediary reflective surface is also a very important factor in non-line-of-sight imaging, which can be generalized to include a full specular reflector like glass (1), a partial specular reflector like ceramic tile (2), and an ideal diffuse surface (3). The study in this paper is mainly carried out on intermediary reflective surface types 1 and 2.
Construction of an optical imaging model of the non-lineof-sight imaging process, which is based on the parameters of the intermediary reflective surface, the parameters of the laser range-gated imaging system, and the properties of the target, is vital for the theoretical analysis and system design of non-line-of-sight imaging based on laser range-gated imaging. This model assumes that the imaging will occur at night or in low-light conditions to reduce the impact of environmental illuminance.
3 Image Contrast Model of Non-Line-of-Sight Imaging As shown in Fig. 3, the laser beam is projected onto an intermediary reflective surface forming the light spot S 1 and experiences first-order reflections; then the first-order reflections are projected onto the target forming light spot S 2 . The distance from the laser to the geometric center of S 1 is l 1 . The distance from the geometric center of S 1 to the geometric center of S 2 is l 2 . The vector tracing the direction of the incident light from S 1 to S 2 makes an angle α with n 1 (the vector normal to S 1 ) and an angle β with n 2 (the vector normal to S 2 ). The optical axis of the laser and the optical axis of the ICCD are assumed to be coincident, and the divergence angle of the laser and the field view angle of the ICCD are also assumed to be coincident. In this section, the laser transmission characteristics (attenuation and distribution) in the non-line-of-sight imaging process will be analyzed.

Pulsed Laser and Range-Gated Imaging System
The illuminance of the pulsed laser can be written as shown in Eq. (1), where PðtÞ denotes the transmission power of the pulsed laser, t is the propagation time, Φ is the energy of a single pulse, t p is the pulse width, Ω laser is the solid angle of the divergence of the laser beam, and θ is half divergence angle of the laser beam. The function rectðtÞ is described by Eq. (3). The detector includes M × N pixels, t g is the gate width, and t d is the strobe gating delay time.

Reflection Characteristics of the Intermediary Reflective Surface
The reflection characteristics (consisting of specular reflection characteristics and diffuse reflection characteristics) of the intermediary reflective surface are among the important factors that can influence the non-line-of-sight imaging results. Without loss of generality, the intermediary reflective surface is assumed to be free of absorption, so that its reflection characteristics can be described by the specular reflectivity ρ s and the diffuse reflectivity ρ d . ρ s is a constant, ρ d obeys Lambertian reflection, and ρ s ¼ 1 − ρ d . The reflectance distribution function is described by with

Target Signal Illuminance and Disturbance
Caused by the Intermediary Reflective Surface under CW-Laser Illumination The illuminance on the target plane S 2 is denoted as E 2 , which consists of two parts: E 2s is caused by the firstorder specular reflection of the laser and E 2d is caused by the first-order forward-scattering. The illuminance on S 1 should be obtained first.
Illuminance on S 1 . According to Eq. (1), the illuminance on S 1 is given by First-order specular reflection. The area illuminated on S 2 by the laser first-order specularly reflected by S 1 can be written as Therefore, E 2s can be calculated as The target distance l is equal to the sum of l 1 and l 2 . If l is much larger than the square root of s 2 , and S 2 is illuminated by the vertically incident laser beam (cos β ¼ 1), Eq. (8) can be further simplified.
First-order forward-scattering. To facilitate the analysis and calculations, Fig. 3 is expanded along the optical path. Assuming that the optical axis of the laser coincides with the optical axis of the ICCD imaging system, a Cartesian coordinate system is established with the geometric center of the lens as the Cartesian coordinate origin. The optical axis is along the z axis, and the y axis and the x axis are, respectively, parallel to the detector array's length and width directions, as is shown in Fig. 4. φ is the angle between the x axis and the projection of n 1 on the xoy plane, and n 2 points in the direction opposite to the direction of the z axis. S 1 and S 2 are, respectively, projections of the laser beam on the intermediary reflective surface and on the target.
According to the geometric relationship, the projection equations for S 1 and S 2 on the xoy plane are, respectively, given by P 1 ðx 1 ; y 1 ; z 1 Þ and P 2 ðx 2 ; y 2 ; z 2 Þ are random points, respectively, on S 1 and S 2 , and d S1 is a microsurface element around P 1 ðx 1 ; y 1 ; z 1 Þ (as shown in Fig. 5). Optical Engineering dE 2d is the illuminance on P 2 ðx 2 ; y 2 ; z 2 Þ caused by d S1 . The illuminance is inversely proportional to the square of the distance as in Eq. (11), where γ 2 is the angle between vectors P 1 P 2 ! and n 1 and η 2 is the angle between vectors P 1 P 2 ! and n 2 ¼ ð0; 0; −1Þ. (12) (13) Therefore, In summary, 3.3.2 Target signal illuminance on the photosensitive surface Assuming that the target plane and the photosensitive surface of the laser range-gated imaging system are conjugate object plane and image plane, and that l is much larger than f 0 ðl ≫ f 0 Þ, the illuminance on the photosensitive surface caused by the scene can be expressed as Eq. (17), where L u ¼ ρ u E 2 ∕π is the luminance on S 2 (ρ u is the reflectance of the target or background, for u ¼ t or b), F ¼ f 0 ∕D 0 denotes the F-number, D 0 is the aperture diameter, and τ o is the spectral transmittance of the objective lens.

Disturbance illuminance on the photosensitive surface caused by scattering of the intermediary reflective surface
The disturbance illuminance on the photosensitive surface caused by scattering of the intermediary reflective surface consists of two parts: E d (third-order forward-scattering) and E back (first-order backscattering). Since the intermediary reflective surface is not on the target plane, each point on S 1 corresponds to a diffuse plaque on the photosensitive surface, and the diffuse plaque can be expressed by a point spread function. A Cartesian coordinate system is established with the geometric center of the lens as a Cartesian coordinate origin. Pð0; 0; l 1 Þ is the intersection of the intermediary reflective surface and the z axis, and P' is the image point of P, as shown in Fig. 6. According to geometrical optics, the center of the diffuse plaque for point ðx 1 ; y 1 Þ on S 1 is given by The radius of the diffuse plaque can be written as where D 0 is the aperture diameter and n ¼ 1 is the refractive index of the object space. The point spread function of the diffuse plaque for point ðx 1 ; y 1 Þ can be expressed as hðx 0 ;y 0 ;x 1 ;y 1 ;l 1 Þ ¼ circ with circðrÞ ¼ circ ffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi First-order backscattering illuminance on the photosensitive surface. According to Eq. (21), the first-order backscattering E back can be expressed as with Ω p ¼ πD 2 0 ∕4ðl 1 þ x 1 · tan αÞ 2 : Ω p is the solid angle defined by the rays emanating from point P 1 ðx 1 ; y 1 ; z 1 Þ to the aperture defined by the lens. In fact, since the distances between the camera and every point on S 1 are not the same, the value of Ω p for each of the points is not the same. As the laser divergence angle θ is very small, an approximate treatment for Ω p is made here.
Third-order forward-scattering illuminance on the photosensitive surface.
In Fig. 5, d S2 is a microsurface element around P 2 ðx 2 ; y 2 ; z 2 Þ. dE 0 1 is the illuminance on P 2 ðx 2 ; y 2 ; z 2 Þ caused by d S2 . The illuminance is inversely proportional to the square of the distance as expressed in Eq. (25), where γ 1 is the angle between vectors P 1 P 2 ! and n 1 and η 1 is the angle between vectors P 1 P 2 ! and n 2 . Then we obtain Since I p can be calculated as the illuminance on the photosensitive surface caused by P 1 ðx 1 ; y 1 ; z 1 Þ can be written as In summary,

Image Contrast Model of Non-Line-of-Sight Imaging
Non-line-of-sight imaging can be achieved with laser rangegated imaging with a pulsed laser; therefore it is necessary to make a distinction between third-order reflection and firstorder backscattering.

Illuminance on the photosensitive surface under pulsed laser illumination
Under pulsed laser illumination, the illuminance on the photosensitive surface caused by a single pulse is a function of time t. (30)

Signal and noise under pulsed laser illumination
Considering the time sequence of the laser range-gated imaging system and the discrete nature of the detector unit, the illuminance on the detector unit ði; jÞ caused by the target and background within a pulse period can be calculated as where a and b are the length and width of the detector unit, respectively. The illuminance on the detector unit ði; jÞ caused by scattering of the intermediary reflective surface can be expressed, respectively, as Optical Engineering We simplify the imaging system noise and environmental noise by combining them into an equivalent incident illuminance E eq , which is used as shown in the following equation: The total disturbance of the non-line-of-sight imaging system can be expressed as

Apparent contrast of the image on a photosensitive surface
The apparent contrast of the image on a photosensitive surface can be described by T sh ði; j; l 1 ; l 2 Þ ¼ where Q tb ðl 1 ; l 2 Þ denotes the averaged signal of the target and background, C 0 ði; jÞ denotes the inherent contrast ratio, and T sh ði; j; l 1 ; l 2 Þ is the contrast transfer function.

Analysis and discussion
In addition to systematic noise, the scattered disturbance illuminance from the intermediary reflective surface is the main factor in image quality degradation.
1. For photoelectric imaging systems based on human eyes, detection and identification of the extended source target depend on the target contour brought out by the apparent contrast of the image. Although people simply believe that 3 to 5% is the contrast threshold for target detection with human eyes, the more effective approaches are based on minimum resolvable contrast (MRC) 8,9 or target task performance (TTP). 10 Enhancing the apparent contrast of the image is conducive to improving the performance of non-line-of-sight imaging system. 2. It is impossible to separate E d ðtÞ from E st ðtÞ because they are synchronized with each other. However, E back may be reduced or eliminated by time gating, as shown in Fig. 7.
• l 2 ≥ c · t p ∕2: E back is separable from E st ðtÞ [as in Fig. 7(a)]. E back may be filtered out completely without reducing the laser illuminance efficiency, and thus, the highest contrast image is obtained.
• l 2 < c · t p ∕2: When the gate opens upon the return of the target signal, part of the first-order backscattering is accepted by the detector, and the image contrast is decreased to some extent. To this end, the image contrast can be improved by adjusting t d [as in Fig. 7(b)].
E back can be totally filtered out by setting the strobe gating delay time to t d ¼ 2l 2 ∕c þ t p . An optimum contrast image can be obtained if the gate width is simultaneously reduced to t g ¼ 2l 2 ∕c. However, utilization rate of the laser illuminance decreases with increase of t d . The laser range-gated imaging system cannot receive the target signal illuminance with t d ¼ 2ðl 1 þ l 2 Þ∕c þ t p , and the efficiency decreases to 0. Since the laser range-gated imaging itself is low-signal-tonoise ratio (SNR) imaging, sacrificing the laser illuminance efficiency significantly reduces the SNR of the image. Therefore, practical application of non-line-of-sight imaging with laser range-gated imaging requires careful balance of the overall system. 4 Imaging Characteristic Analysis of Non-Line-of-Sight Imaging Based on the image contrast model of non-line-of-sight imaging constructed above, this section analyzes the nonline-of-sight imaging results with different intermediary reflective surfaces. Typical parameters of non-line-of-sight imaging with a laser range-gated imaging system are shown in Table 1. The imaging depth of field is 36 m with a laser pulse width of 120 ns and a gate width of 120 ns; 40 laser pulses are received by the photodetector within a frame when the imaging frame rate is 25 fps and the laser pulse frequency is 1000 Hz, so that a laser illuminance equivalent to 4.8 μJ is received by the photodetector per frame.

Impacts of Reflection Characteristics of the Intermediary Reflective Surface on Non-Line-of-Sight Imaging
The impacts of the reflection characteristics of the intermediary reflective surface on non-line-of-sight imaging are considered with l 1 ¼ 5 m, l 2 ¼ 20 m, and α ¼ 45 deg.
Since l 2 ≥ c·t p ∕2 ¼ 18 m, the target signal illuminance and first-order backscattering are separable. Figure 8 shows the curves of the target signal illuminance E st , the first-order backscattering E back , and the third-order forward-scattering E d for different values of the diffuse reflectivity ρ d of the intermediary reflective surface with a CW laser. Since E back and E d are unevenly distributed over the imaging field, the E back and E d in Fig. 8 are the respective means of E back and E d in the center area of the imaging field. The curves in Fig. 8 indicate the following conclusions: 1. Under a CW laser, the target signal illuminance E st monotonically decreases with ρ d , and first-order backscattering E back monotonically increases with ρ d . E st is larger than E back only when ρ d is <0.05 (i.e., diffuse reflection takes up only 5% of the total reflection illuminance, and the specular reflection illuminance dominates), that is to say that non-line-of-sight imaging can be realized with only a CW laser when the intermediary reflective surface has very strong specular reflection characteristics. This conclusion indicates that practical realization of non-line-of-sight imaging must use a pulsed laser range-gated imaging mode in order to filter out E back . In the instance indicated in Fig. 8, E back can be totally filtered out without reducing the laser illuminance efficiency. 2. The target signal illuminance E st spans more than three orders of magnitude from 4.6 × 10 −3 W∕m 2 for a completely specular reflection (ρ d ¼ 0) to 5.1 × 10 −7 W∕m 2 for a nearly completely diffuse reflection (ρ d ¼ 0.99). E d is several orders of magnitude smaller than E st , and E d is two orders of magnitude smaller than E st even when ρ d is equal to 0.99. Figure 9 shows that the distribution of E d on the photodetector for l 1 ¼ 5 m, l 2 ¼ 20 m, φ ¼ 0 deg, θ ¼ 2 deg, and ρ d ¼ 0.5 is within the same order of magnitude for α (the tilt angle of the intermediary reflective surface) equals 20 or 60 deg. The image quality is severely disturbed by quantum noise when the target signal illuminance is too low, so ρ d ≤ 0.7 is selected as an effective non-line-of-sight imaging condition. E st with ρ d ¼ 0.7 is one order of magnitude smaller than E st with ρ d ¼ 0. E d is always two orders of magnitude smaller than E st when ρ d changes from 0 to 0.7. In view of the above discussion, E d is often negligible.

Impacts of the Location of the Intermediary
Reflective Surface and the Gate Width on Non-Line-of-Sight Imaging The parameter t g is chosen to be equal to t p to fully use the laser illuminance and to reduce the noise for a typical laser range-gated imaging system, in which case the imaging depth of field is c·t p . The gate opens when the echo signal reaches the detector, so that the gate delay time t d equals 2ðl 1 þ l 2 Þ∕c. The following discussion is based on the assumption that Q eq is equal to 0.  Figure 10 shows a plot of the image contrast C versus distance l 2 with l 1 ¼ 5 m and t d ¼ 2ðl 1 þ l 2 Þ∕c. The plot shows that C first decreases with l 2 and then increases with l 2 to a certain constant value at l 2 ¼ 18 m when E back is totally separated from E st and that finally C is nearly fixed with increasing l 2 . Figure 11 shows a plot of the image contrast C versus distance t d with l 1 ¼ 5 m and l 2 ¼ 10 m. The plot shows that C increases with t d and that C obtains its maximum value when E back is totally filtered out at t d ¼ 140 ns.

Conclusions
This paper describes the construction of an image contrast model of non-line-of-sight imaging with a laser rangegated imaging system on the basis of the modeling of an intermediary reflective surface. The image contrast model analyzes three factors affecting the image contrast of nonline-of-sight imaging with different intermediary reflective surfaces, where the three factors are target signal illuminance, first-order backscattering, and third-order forwardscattering. A non-line-of-sight imaging experiment simulation was carried out with the parameters of an actual laser range-gated imaging system. The simulation results show that (1) the first-order backscattering is the dominant disturbance to non-line-of-sight images and can be effectively filtered out by laser range-gated imaging and (2) the effect of the third-order forward-scattering on non-line-of-sight images is often negligible since the third-order forward-scattering is insignificant compared to the target signal illuminance in most cases. The image contrast model of nonline-of-sight imaging with a laser range-gated imaging system gives insights into the theoretical analysis and system design of non-line-of-sight imaging.
The simulation results also show that the reflection characteristics of the intermediary reflective surface have a significant effect on non-line-of-sight imaging. The bidirectional reflectance distribution function (BRDF) is a more comprehensive description of surface reflection characteristics; thus, BRDF modeling of typical construction material surfaces is a direction for further research. Meanwhile, since non-line-of-sight imaging with laser range-gated imaging system is usually low-SNR imaging, estimating the range of non-line-of-sight imaging using MRC or TTP is another direction for further research.