Review of three-dimensional imaging of dynamic objects by parallel phase-shifting digital holography

Takahito Fukuda; Yasuhiro Awatsuji; Peng Xia; Takashi Kakue; Kenzo Nishio; Osamu Matoba

doi:10.1117/1.OE.57.6.061613

6 April 2018 Review of three-dimensional imaging of dynamic objects by parallel phase-shifting digital holography

Takahito Fukuda, Yasuhiro Awatsuji, Peng Xia, Takashi Kakue, Kenzo Nishio, Osamu Matoba

Author Affiliations +

Optical Engineering, Vol. 57, Issue 6, 061613 (April 2018). https://doi.org/10.1117/1.OE.57.6.061613

Abstract

Parallel phase-shifting digital holography is a powerful technique for recording motion picture of holograms with an image sensor and numerically reconstructing the motion picture of the high-quality three-dimensional (3-D) images of the object in a computer. This technique provides high-quality images of light intensity and phase of the object at a time instant at arbitrary depth position. Therefore, the technique achieves 3-D motion-picture imaging of a dynamic object. The technique provides not only large depth of field and high temporal resolution at the same time but also a motion picture of an invisible object. It is difficult for other 3-D imaging techniques to achieve these features of parallel phase-shifting digital holography. The authors review two experiments demonstrating the 3-D imaging by the technique and the features of the technique. One experiment demonstrated the motion-picture 3-D imaging of a minute crystal, sinking down in solution, by refocusing the amplitude images of the crystal. The result was obtained for the first time, to the best of our knowledge. The other demonstrated the motion-picture 3-D imaging of refractive indices of dynamic invisible gas flow by applying the Abel inversion to the phase images of the flow obtained by the technique.

1. Introduction

In recent decades, three-dimensional (3-D) imaging technique has been studied in many fields, such as bioimaging,¹^–⁸ particle measurement,⁹^–¹³ flow measurement,¹⁴^–¹⁹ shape measurement,²⁰ deformation measurement,²¹^–²⁵ and so on. Many 3-D imaging techniques such as stereoscopic imaging²⁶ and light-field imaging²⁷ have been studied. Especially, holography²⁸ is an attractive 3-D imaging technique, because the technique obtains not only the light intensity but also the phase information of the object. This is because holography can record the wavefront of a light wave from an object. In this technique, the interference fringe image generated by two light waves is recorded on a holographic plate. One wave is reflected by or transmitted through an object, and the other wave is incident to the holographic plate without illuminating the object. The former and the latter are called as the object wave and the reference wave, respectively. The holographic plate on which the interference fringe image was recorded is called the hologram. When the plate is illuminated by the light wave having the same wavelength and incident angle as the reference wave, the object wave is reconstructed and the 3-D image of the object can be observed. Although holography provides a high-quality 3-D image of an object, it is difficult for the holographic plate to record a motion picture of a hologram. This is because a holographic plate requires a burdensome process such as the chemical development of the holographic plate after recording the holograms, in general. To avoid the burden, digital holography has been studied.²⁹ Digital holography records an interference fringe image as digital data using an image sensor such as a charge-coupled device or complementary metal-oxide semiconductor image sensor, instead of a holographic plate. Digital holography reconstructs the amplitude and phase of an object by numerical process and displays the 3-D image in a computer. In particular, parallel phase-shifting digital holography was expected to obtain high-quality 3-D motion-picture of an object.³⁰ Owing to the capability, many papers have reported high-speed motion-picture 3-D measurement of dynamic biological specimen, invisible air flow, sound-field, wavefront from an optical device, vibration of a loudspeaker, light propagation, and so on.³¹^–³⁷

1.1.

Digital Holography

Because the amplitude and phase distributions of the object wave at arbitrary depth position can be calculated, digital holography can numerically reconstruct the 3-D image of the object. Owing to the acquisition capability of quantitative complex amplitude of the light wave from an object, the technique has been actively studied for 3-D shape measurement, 3-D motion measurement, phase measurement, and so on. However, an image sensor is not able to record the same interference fringes as a holographic plate completely, because the pixel pitch of the image sensor is too large to resolve fine interference fringes. Then, in-line digital holography, in which object wave and reference wave are on-axis, is often used because in-line digital holography enables us to record more information of the interference fringes and covers a larger measurement range and provides higher resolution than off-axis digital holography. However, there is a problem in in-line digital holography: undesired images, the zeroth-order diffraction wave and conjugate image are superimposed on the image of the object. As a result, the quality of the reconstructed image is degraded.

1.2.

Phase-Shifting Digital Holography

To reconstruct the image free from the undesired images, phase-shifting digital holography was proposed.³⁸ Phase-shifting digital holography sequentially records multiple holograms with phase-shifting of the reference wave. From the recorded phase-shifted holograms, the complex amplitude free from the undesired images is numerically reconstructed in a computer as the following procedure. Here, we define the complex amplitude distribution of an object wave on an image sensor plane as $U_{o} (x, y) = A_{o} (x, y) \exp [j ϕ (x, y)]$ . $A_{o} (x, y)$ is the amplitude, $ϕ (x, y)$ is the phase, and j is an imaginary unit. We assume that the wave propagates along the $z$ axis. Although four-step phase-shifting digital holography is often employed, parallel phase-shifting digital holography recording fewer phase-shifted holograms has been proposed so as to reconstruct higher-quality image than that of four-step phase-shifting digital holography.³⁹^,⁴⁰ We now describe two-step phase-shifting digital holography,⁴¹ for example. Two holograms $I_{0} (x, y)$ , $I_{π / 2} (x, y)$ with phase difference $π / 2$ are recorded and then complex amplitude of the object wave at the image sensor plane is given as

Eq. (1)

U_{o} (x, y) = \frac{[I_{0} (x, y) - a (x, y)] + j [I_{π / 2} (x, y) - a (x, y)]}{2 A_{r} (x, y)},

Eq. (2)

a (x, y) \equiv A_{o}^{2} (x, y) + A_{r}^{2} (x, y) = \frac{v (x, y) - \sqrt{v^{2} (x, y) - 2 w (x, y)}}{2},

Eq. (3)

v (x, y) \equiv [I_{0} (x, y) + I_{π / 2} (x, y)] + 2 A_{r}^{2} (x, y),

Eq. (4)

w (x, y) \equiv [I_{0}^{2} (x, y) + I_{π / 2}^{2} (x, y)] + 4 A_{r}^{4} (x, y),

where

A_{r}^{2} (x, y)

is the intensity distribution of the reference wave.

U_{o} (x, y)

, the complex amplitude distribution of the object wave at the image sensor plane

z = 0

, is given by Eq. (1). Then,

U (x, y, z)

, the complex amplitude distribution of the object wave at arbitrary depth

z

, is obtained by calculating the light propagation in the depth direction. Light propagation can be calculated by several methods such as the Fresnel transformation or the angular spectrum method.⁴² Here, we describe the angular spectrum method as follows:

Eq. (5)

U (x, y, z) = {FT}^{- 1} {FT [U_{o} (x, y)] \exp [- j z (k^{2} - k_{x}^{2} - k_{y}^{2})]},

where

k = 2 π / λ

,

λ

,

k_{x}

, and

k_{y}

are the wave number, the wavelength, and the wave numbers in the

x

and

y

directions, respectively. FT and

{FT}^{- 1}

are the Fourier transform operation and the inverse Fourier transform operation, respectively.

Indeed, phase-shifting digital holography can provide high-quality 3-D imaging of an object free from the undesired images. However, the technique requires sequential recording of multiple holograms. Therefore, the technique is useless for a dynamic object, much less recording motion picture of holograms of the dynamic object. To cope with a dynamic object, parallel phase-shifting digital holography was proposed.

1.3.

Parallel Phase-Shifting Digital Holography

Parallel phase-shifting digital holography is a technique that implements phase-shifting digital holography with a single-shot recording of multiple phase-shifted holograms, which are required for phase-shifting digital holography, using a single image sensor. Figure 1 shows a schematic of the principle of parallel phase-shifting digital holography. This figure shows the example where the number of the steps of the phase-shifting is $N$ . In this technique, an image sensor records multiple phase-shifted holograms as a single hologram. The holograms are recorded by each pixel of the sensor based on a space-division multiplexing technique. Two methods have been proposed to derive the complex amplitude of the object. One method interpolates the blanked pixels of each hologram³⁰ and the other conducts the calculation used in the phase-shifting interferometry using neighboring pixels.⁴³^,⁴⁴ In this paper, we describe the interpolation method often used. Pixels of each phase-shifted hologram are extracted from the recorded single hologram. Each blanked pixels of each of the phase-shifted holograms is interpolated using the neighboring pixels. After the interpolation, the multiple holograms required for phase-shifting digital holography are generated. Then, the complex amplitude at the image sensor plane is obtained by the calculation used in the phase-shifting interferometry, and the complex amplitude of the object wave at arbitrary depth is obtained by calculation of light propagation.

Fig. 1

Schematic of the principle of parallel phase-shifting digital holography in which the number of the steps of the phase-shifting is $N$ .

Owing to the capability of simultaneous recording of multiple holograms, parallel phase-shifting digital holography can record motion-picture 3-D images at the same frame rate as that of the image sensor. In other 3-D imaging techniques, there are trade-off relations between depth of field and temporal resolution or between depth resolution and lateral resolution. In contrast, there are not the relations in parallel phase-shifting digital holography. Furthermore, the technique can obtain a motion picture of an invisible object, because the technique can obtain motion picture of the phase distribution of the object. Both of these are the advantageous features of parallel phase-shifting digital holography compared with other 3-D imaging techniques.

In this paper, we review two experimental demonstrations of the motion-picture 3-D imaging of dynamic objects using parallel phase-shifting digital holography in detail. One used the amplitude distribution and the other used the phase distribution. These distributions of dynamic objects were obtained by the technique. Both demonstrations show the features of the technique.

2. Three-Dimensional Imaging of a Dynamic Object by Parallel Phase-Shifting Digital Holography

2.1.

Using the Amplitude Image

In general, the depth of field is quite small in microscopic measurement. Then, we experimentally demonstrated 3-D measurement of a dynamic minute specimen using the amplitude obtained by parallel phase-shifting digital holography and display the measurement result in 3-D space.⁴⁵

2.1.1.

Methods

Figure 2 shows the schematic of the optical setup of the experimental system. To implement the parallel phase-shifting, a microphase-shifting array,³⁰^,³⁹^,⁴⁰^,⁴³^,⁴⁶ a spatial light modulator,⁴⁷ and a microarray device such as a microwave plate array or a micropolarizer array³¹^,³²^,³⁶^,³⁷^,⁴⁸^–⁵⁶ have been proposed. This system is a microscope system based on the implementation of parallel phase-shifting digital holography using polarization.⁴⁸^,⁵⁶ A light wave is split into two waves by the polarizing beam splitter. The polarization state of the two waves is orthogonal to each other. One wave transmits through a specimen and becomes the object wave, then this wave is combined with the other wave, the reference wave. The half-wave plate is used to adjust the intensity ratio of the two waves adequately. By passing through the quarter-wave plate in front of the camera, the two waves are transformed into clockwise and counterclockwise circularly polarized light waves, respectively. The camera has a micropolarizer array device consisting of the blocks consisting of $2 \times 2$ cells of micropolarizer, as shown in Fig. 2. Intensities of four linearly polarized light beams are detected by each $2 \times 2 pixels$ of the image sensor after passing through the array device, whose polarization directions are 0 deg, 45 deg, 90 deg, and 135 deg. Therefore, the polarization-imaging camera is capable of capturing two-dimensional (2-D) distribution of polarization state of an incident light into the image sensor. Thus, four interference fringe images with phase differences 0, $π / 2$ , $π$ , and $3 π / 2$ are recorded by each of the $2 \times 2 pixels$ of the camera. Consequently, four phase-shifted holograms required for four-step phase-shifting digital holography are recorded by space-division multiplexing technique with a single-shot exposure. We reconstructed the complex amplitude distribution of the object by applying the image-reconstruction algorithm of two-step parallel phase-shifting digital holography to the recorded four holograms.⁴⁹

Fig. 2

Schematic of the optical setup of the microscope system based on parallel phase-shifting digital holography. PBS, polarizing beam splitter; HWP, half-wave plate; QWP, quarter-wave plate; PIC, polarization-imaging camera; MO, microscope objective; TL, tube lens.

We set a magnification optical system to measure a minute specimen, and employed an afocal system as the magnification optical system. The afocal system consists of two lenses with the focal lengths of $f_{1}$ and $f_{2}$ . The lenses are separated by a distance of $f_{1} + f_{2}$ . The lateral magnification $M_{lat}$ and the longitudinal magnification $M_{long}$ depend only on both the focal lengths and given as

Eq. (6)

M_{lat} = f_{2} / f_{1},

Eq. (7)

M_{long} = {(f_{2} / f_{1})}^{2} .

The longitudinal magnification $M_{long}$ is equal to the square of the lateral magnification $M_{lat}$ , as shown in Eqs. (6) and (7). Therefore, when the microscope system consists of a microscope objective with a short focal length and a tube lens with a longer focal length, the depth of field is quite small. Consequently, it is difficult for a usual microscope to observe a specimen moving in the depth direction. In contrast, parallel phase-shifting digital holography is capable of motion-picture measurement of a dynamic specimen moving in 3-D, owing to the capabilities of the single-shot recording of holograms and digital refocusing at arbitrary depth. Indeed also light-field imaging achieves 3-D imaging with an image sensor and with a single-shot exposure, but there is a trade-off relation between lateral resolution and depth resolution and this technique cannot obtain the phase image of an object. In contrast, there is not the relation in parallel phase-shifting digital holography and the technique can obtain the phase image.

Furthermore, we employed an inverted system as the magnification optical system, whose specimen is illuminated from above. This is because an inverted system is suitable for demonstrating motion-picture 3-D imaging of a minute object sinking down in an aqueous solution in the depth direction.

2.1.2.

Experimental setup

As shown in the dashed line in Fig. 2, the inverted afocal magnification optical system was constructed. The light wave for illuminating the specimen is reflected upward then horizontally by mirrors. The wave is reflected downward by another mirror and directed on a specimen. The wave transmits the specimen and becomes the object wave. After that, the object wave is finally reflected horizontally by the other mirror and recorded by a polarization-imaging camera.

Figure 3 shows the photograph of the optical system shown in Fig. 2. A $Nd : {YVO}_{4}$ laser operated at 532 nm was used as the light source. A Photron FASTCAM-SA2-P whose pixel pitch is $10 μ m$ was used as the polarization-imaging camera. This high-speed camera can record motion picture at the frame rate up to 86.2 kilo frames per second (fps). The inverted afocal magnification optical system consisted of a microscope objective with 16.6-mm focal length and a convex lens with 300 mm focal length. Equations (6) and (7) give the lateral and longitudinal magnifications 18.1 and 328. We use the criterion of resolution defined by Abbe, which is used when specimens are illuminated by a coherent parallel light wave.⁵⁷ Then, the lateral resolution $δ = λ / NA$ is calculated and $2.1 μ m$ is obtained.

Fig. 3

Photographs of the constructed optical system of the microscope system based on parallel phase-shifting digital holography. (a) Top view of the whole system and (b) side view of the magnification optical system shown in the dashed line square in (a). PBS, polarizing beam splitter; HWP, half-wave plate; QWP, quarter-wave plate; PIC, polarization-imaging camera; P, polarizer; MO, microscope objective; TL, tube lens.

2.1.3.

Results

First, we recorded salt crystals of $10 μ m$ order precipitated at the bottom of a container filled with solution of salt, then reconstructed the complex amplitude of the crystal. Figure 4 shows the experimental result. Figures 4(a) and 4(c) show the amplitude image and the phase image reconstructed by parallel-phase-shifting digital holography. Figures 4(b) and 4(d) show the amplitude image and the phase image reconstructed by the in-line digital holography without the phase-shifting, for comparison. In the images reconstructed by the in-line digital holography without phase-shifting, the edges of the crystal are seen double, and the contrast was emphasized undesirably, as shown in Figs. 4(b) and 4(d). This is because the undesired images were superimposed on the desired images and interfered with the desired images. In contrast, parallel-phase-shifting digital holography reconstructed clear images free from the unwanted images. Thus, it was demonstrated that parallel phase-shifting digital holography is capable of high-quality imaging of a minute specimen in microscopic measurement.

Fig. 4

Reconstructed images of salt crystals: (a) amplitude and (c) phase images reconstructed by parallel-phase-shifting digital holography, (b) amplitude and (d) phase images reconstructed by the in-line digital holography without phase-shifting.

Next, we recorded an alum crystal sinking down in the solution of alum with the same optical system and displayed the result in 3-D space. The frame rate, the shutter speed, and the total recording time were 60 fps, 10 ms, and 18 s, respectively. Figures 5 and 6 show the results. Each of the images is the amplitude image extracted from the reconstructed motion picture every 160 frames in Figs. 5 and 6, respectively. Thus, the time interval of the images is 2.67 s. Figure 5 shows the images refocused on the fixed plane where the crystal was located when the recording of the motion picture was finished. The images were not refocused for most of the time on the crystal sinking down in the depth direction, which is the same as the image observed by a usual microscope. Therefore, it is difficult to specify the position and the direction of the crystal. In contrast, Fig. 6 shows the images refocused at all times on the plane where the center of the crystal was. We refocused on a corner of the crystal and defined the depth position when we refocused on the corner as the depth position where the reconstructed image of the corner has the sharpest edge, at each 160 frames. Then, we defined the center of the crystal as the point of an intersection between 3-D diagonal lines of the refocused corners, at each 160 frames. These depth positions were calculated from the propagation distance obtained by calculation of the light propagation of the object wave. The object wave was refracted by the aqueous solution and magnified by the magnification optical system. To calculate the exact position of the crystal, it is necessary to know the refractive index of the aqueous solution and to compensate the refraction. We measured the refractive index of the solution and obtained the value of 1.36. The depth positions $z$ were calculated from the propagation distance, the refractive index of 1.36, and the longitudinal magnification of 328. These positions are (a) $1020 μ m$ , (b) $997 μ m$ , (c) $871 μ m$ , (d) $655 μ m$ , (e) $334 μ m$ , (f) $87 μ m$ , (g) $- 66 μ m$ , and (h) $- 66 μ m$ . Here, the $z$ axis was taken in the counter direction of the direction in which the crystal was sinking down, and $z = 0$ indicates the depth position where the center of the crystal was located when we started the recording of motion picture of holograms. Figure 6 shows the images refocused at all time on the crystal sinking down in the depth direction and also shows the crystal was rotating while sinking. As shown in Fig. 6, we successfully demonstrated quantitative 3-D imaging of the crystal, whose shape and size were approximately regular octahedron and about $25 μ m$ in a side, respectively, sinking in the depth direction.

Fig. 5

Reconstructed amplitude images refocused on the fixed plane. The images were refocused on the plane where the crystal was at (h). Each time interval of (a)–(h) is 2.67 s.

Fig. 6

Reconstructed amplitude images of minute alum crystal in focus at all time. The crystal was tracked and the images were refocused at all time on the plane where the crystal was. Each time interval of (a)–(h) is 2.67 s. Positions of the crystal in depth direction are $z =$ (a) $1020 μ m$ , (b) $997 μ m$ , (c) $871 μ m$ , (d) $655 μ m$ , (e) $334 μ m$ , (f) $87 μ m$ , (g) $- 66 μ m$ , and (h) $- 66 μ m$ .

Furthermore, we calculated the 3-D displacement of the crystal. Because the lateral positions were also calculated when the depth positions were calculated, the 3-D positions were found. Figure 7 shows the 3-D trajectory of the crystal. The trajectories in (a) 3-D, (b) the $x - z$ plane, i.e., top view, (c) the $y - z$ plane, and (d) the $x - y$ plane are shown. Each of the open circles shows the position of the center of the crystal at each 20 frames, i.e., each 0.333 s, and the dashed curve shows the trajectory of the center. The $x - y$ plane shows the lateral plane of the reconstructed image, and the original position shows the lateral position of the crystal when the recording was started. As shown in Fig. 7, we successfully demonstrated quantitative 3-D imaging of the minute crystal sinking in the solution.

Fig. 7

3-D tracking of a minute alum crystal sinking down in aqueous solution. Trajectories in (a) 3-D, (b) $x - z$ plane, (c) $y - z$ plane, and (d) $x - y$ plane. The closed square shows the position when the recording of motion picture of holograms was started, and the closed triangle shows the position when the recording was finished.

2.2.

Using the Phase Image

In this section, we review the motion-picture 3-D imaging of a dynamic invisible object using the phase image of the object obtained by parallel phase-shifting digital holography.⁵⁸