1.

Introduction

Holography¹ involves recording and reconstruction of three-dimensional (3-D) images using interference and diffraction of light. Holography records 3-D images of an object as interference fringe patterns called holograms. Since holography can satisfy all of the physiological factors required for 3-D recognition, such as binocular disparity, motion parallax, focus adjustment, and convergence, they can display 3-D images without causing eye fatigue.

Although classical holography can reconstruct high-quality 3-D images using high-resolution holographic recording materials, dynamic objects are difficult to record and reconstruct because rapid rewriting of these materials is difficult. Electroholography has been proposed to address this challenge.² Because electroholography uses a spatial-light modulator (SLM), which can dynamically switch displayed patterns, to display holograms, electroholography can reconstruct 3-D motion pictures. Here, hologram patterns need to be obtained on a computer to display holograms using an SLM. Computer-generated holograms (CGHs)^3,4 can be produced by numerically simulating the propagation of light from the 3-D object to the hologram plane with diffraction calculations. Several methods are available to calculate CGHs, such as the point-cloud method, the polygon method, and the multiview-image method, but they all entail computational costs and complexity that render dynamic reconstruction of 3-D objects impractical. CGH calculation must be accelerated or made more efficient to make real-time electroholographic displays feasible. With the point-cloud method, CGHs can be calculated using graphics processing units (GPUs)^5–9 because the calculations can be parallelized because of the independence of each pixel in the holographic plane. However, for the point-cloud method, real-time calculation for a high-resolution CGH of a large 3-D object is computationally complex. Hence, reducing the computational complexity of the CGH calculations in the point-cloud method could yield a method rapid, real-time CGH calculation.

A recurrence-relation algorithm^10–13 can reduce computational complexity by using addition instead of multiplication operations in its calculations. This algorithm exploits the phase differences between neighboring pixels in the hologram plane and is well suited to hardware implementations of CGH calculations. Some researchers have developed special-purpose computers, referred to as “holographic reconstruction: HORN,” to distribute the calculations. The recurrence-relation algorithm was adopted by HORN system, and CGH calculations were successfully obtained by implementing a pipeline structure with parallel computers, such as field-programmable gate arrays.^14–17 The latest version of the HORN system, the eight-board HORN-8 cluster, was used to accelerate the CGH calculations by more than 1000 times compared to the speed achieved with the use of a central-processing unit (CPU), making optimal use of four cores.¹⁷

A patch-model algorithm^18,19 also exploits the recurrence relations between neighboring pixels. Although the patch-model algorithm can be used to effectively accelerate the CGH calculations when there is a large number of point-light sources, the error caused by approximation becomes large when the distance between a 3-D object and the CGH plane is large. A look-up table (LUT) algorithm can also be used to reduce computational complexity.^20–28 The LUT algorithm precalculates values of complex amplitude and stores them in an LUT. Although storing precalculation results in the LUT requires large memory allocation, this strategy can significantly accelerate CGH calculations and can be combined with other approaches. An algorithm based on image-type holograms has successfully reduced computational complexity.^29,30 This algorithm calculates CGH patterns on only a small region of the CGH plane because the 3-D object is placed close to the CGH plane. The wavefront-recording-plane (WRP) algorithm^31–37 also requires that the 3-D object be placed near the WRP. Although the WRP algorithm requires diffraction calculations for the region between the WRP and the CGH plane in addition to CGH calculations based on the point-cloud method, the diffraction calculations can be accelerated using fast Fourier transforms (FFTs). Because the computational complexity of the FFT can be expressed as $O(N\log N)$, where $N$ indicates the sampling number and is a power of 2, the WRP algorithm can be used to drastically accelerate the CGH calculations when the CGH resolution is large. The acceleration ratio in the WRP algorithm decreases as the distance from the 3-D object increases.

An algorithm based on point symmetry of a zone plate, which indicates a CGH generated from a single-point light source, can reduce the computational complexity.^38,39 An adaptive point-spread spherical wave synthesis algorithm^40,41 accelerates CGH calculations by reducing the number of point-light sources using data-compression ideas from graphics processing. An algorithm based on the sparsity of hologram patterns⁴² can accelerate CGH calculations by using sparse FFTs whose computational complexity is lower than that of standard FFTs. A wavelet shrinkage-based superposition algorithm accelerates CGH calculations by approximating point-spread functions with a representative sample of wavelet coefficients.⁴³ Table 1 summarizes the advantages and disadvantages of several representative CGH-calculation algorithms used in the point-cloud method.

Table 1

Advantages and disadvantages of representative CGH-calculation algorithms for the point-cloud method.

In this paper, we review two strategies for achieving electroholographic display systems that can project aerial 3-D images in real time. Both strategies are based on the point-cloud method. Although the point-cloud method requires a huge number of point-light sources to obtain a realistic model of a 3-D object, the method is well suited to hardware implementation as the CGH-calculation processes can be parallelized with the pipeline structure. Miniaturization or modularization of the CGH calculation system is required for applications to head-mounted displays and other mobile devices. For example, our research group aims to create a chip dedicated for electroholography to upgrade the HORN system.¹⁷ Hence, we adopted the point-cloud method in the two real-time electroholographic display systems described herein. The first reduces the computational load required for each CGH, and the second parallelizes the CGH calculations using a GPU instead of the HORN system.

2.

Electroholography and Computer-Generated Holograms

2.1.

Electroholography

Figure 1 is a schematic of the optical setup used for electroholography. Light from the optical source passes through the beam expander and illuminates the SLM. The SLM diffracts the light according to the pattern of the hologram. The first-order diffracted light is referred to as an object wave and corresponds to the light emitted from the 3-D object while recording the hologram. An observer sees a 3-D object when receiving the object waves in a sequence. Although Fig. 1 shows a monochromatic reconstruction system for simplicity, full-color reconstruction is possible with multiple optical sources of red, green, and blue light. Many studies have been published regarding color electroholography using several SLMs^44–49 or using a single SLM with space-division multiplexing,^50,51 depth-division multiplexing,^52,53 and time-division multiplexing.^54–59

Fig. 1

Schematic of an optical setup for electroholography.

2.2.

Computer-Generated Holograms

A CGH^3,4 is a hologram that is generated by numerical simulation of light diffraction and interference patterns. CGHs can be calculated via methods, such as the point-cloud method, the polygon method, and the multiview-image method.

2.2.1.

Point-cloud method

Figure 2 is a schematic of the point-cloud method. A 3-D object is represented as an aggregate of point-light sources, known as a point cloud. We denote the coordinates of the $l$’th point-light source ${\mathrm{P}}_l$ and the corresponding pixel on the hologram plane by ${\mathrm{P}}_l(x_l,y_l,z_l)$ and $\mathrm{H}(x_H,y_H,0)$, respectively. Then, $u_l(x_H,y_H,0)$ indicates the complex amplitude of the light emitted from ${\mathrm{P}}_l$ at H:

Eq. (1)

$$u(x_H,y_H,0)=\frac{A_l}{r_l}\exp \{-i\frac{2\pi }{\lambda }\mathrm{sgn}(z_l)r_l\},$$

Eq. (2)

$$r_l=\sqrt{{(x_l-x_H)}^2+{(y_l-y_H)}^2+z_l^2}.$$

Fig. 2

Schematic of the point-cloud method.

Here, $A_l$, $i$, $\lambda$, and $\mathrm{sgn}(R)$ indicate the amplitude of the light emitted from ${\mathrm{P}}_l$, the imaginary component, the wavelength of light, and the signum function of a real number $R$, respectively. We can calculate $h(x_H,y_H,0)$, the complex amplitude formed by the 3-D object at H, by aggregating $u_l(x_H,y_H,0)$ from all point sources of light:

Eq. (3)

$$h(x_H,y_H,0)=\sum _{l=1}^L\frac{A_l}{r_l}\exp \{-i\frac{2\pi }{\lambda }\mathrm{sgn}(z_l)r_l\},$$

where $L$ is the number of point-light sources. The hologram pattern can finally be calculated from $h(x_H,y_H,0)$. Since commercially available SLMs can modulate only intensity or phase of light and not complex amplitude, amplitude-modulation or phase-modulation holograms need to be generated. We represent the amplitude-modulation hologram $h_{\text{amplitude}}(x_H,y_H,0)$ and phase-modulation type hologram $h_{\text{phase}}(x_H,y_H,0)$ with the following equations:

Eq. (4)

$$h_{\text{amplitude}}(x_H,y_H,0)=\mathrm{Re}[h(x_H,y_H,0)],$$

Eq. (5)

$$h_{\text{phase}}(x_H,y_H,0)=\arg [h(x_H,y_H,0)].$$

Here, $\mathrm{Re}[C]$ and $\arg [C]$ indicate the real part and the argument of a complex number $C$, respectively. Since Eq. (1) is independently calculated for each pixel on the hologram plane and each point source of light from the 3-D object, we can accelerate the calculation using parallel processing with multiple cores in CPU or a GPU. We can also accelerate CGH computations by using fast algorithms for the point-cloud calculations.^{6,7,10–13,18–43}

2.2.2.

Polygon method

Figure 3 is a schematic of the polygon method.^60–64 The polygon method expresses a 3-D object as an aggregate of planar polygons. We can obtain $h(x_H,y_H,0)$ by accumulating the light diffracted from each polygon to the hologram plane. Light diffraction is calculated by the Fresnel diffraction method or the angular-spectrum method. We assume that the complex amplitudes of the source plane and the destination plane are denoted as $u_s(x_s,y_s,0)$ and $u_d(x_d,y_d,z)$, respectively. The Fresnel diffraction method⁶⁵ can be expressed by

Eq. (6)

$$u_d(x_d,y_d,z)=\mathrm{IFT}[\mathrm{FT}[u_s(x_s,y_s,0)]\mathrm{FT}[g(x_s,y_s,z)]],$$

Eq. (7)

$$g(x_s,y_s,z)=\exp [i\frac{\pi }{\lambda z}\{{(x_d-x_s)}^2+{(y_d-y_s)}^2\}].$$

Fig. 3

Schematic of the polygon method.

Here, $\mathrm{FT}[C]$ and $\mathrm{IFT}[C]$ indicate the Fourier transform and inverse Fourier transform of a complex number $C$, respectively. The angular-spectrum method⁶⁵ can be expressed by

Eq. (8)

$$u_d(x_d,y_d,z)=\mathrm{IFT}[\mathrm{FT}[u_s(x_s,y_s,0)]\exp (i2\pi z\sqrt{\frac{1}{{\lambda }^2}-f_x^2-f_y^2}\left)\right],$$

where $f_x$ and $f_y$ indicate the spatial frequencies on the $x$ and $y$ axes, respectively. Since Eqs. (6) and (8) are based on Fourier transforms, we can accelerate the calculations using FFTs. However, the Fresnel diffraction method and the angular-spectrum method can be used only when the sampling pitches of the source and destination planes are equal and the planes are parallel. To overcome this issue, several algorithms have been proposed: the shifted Fresnel diffraction method,⁶⁶ aliasing-reduced shifted and scaled Fresnel diffraction method,⁶⁷ scaled angular-spectrum method,^68,69 etc. In addition, by applying a nonuniform sampling method^70–72 to the Fresnel diffraction and angular-spectrum calculations, we can calculate diffractions between planes that were not uniformly sampled.

2.2.3.

Multiview-image method

Figure 4 shows a schematic of the multiview-image method. This method generates a hologram pattern from a series of 2-D images captured from different viewing positions on the hologram plane. Although the multiview-image method can be realized with holographic stereogram^73–76 or ray-wavefront conversion^77–84 approaches, they both acquire the complex amplitudes of a 3-D scene from the 2-D images using Fourier transforms. Hence, CGH calculations in the multiview-image method can be accelerated with FFT. Ichihashi et al.⁷⁹ succeeded in reconstructing a 3-D scene in real time from CGHs with $7680\times 4320\text{pixels}$ at $12\text{frames}/\mathrm{s}$ using ray-wavefront conversion approach.

Fig. 4

Schematic of the mulitview-image method.

2.2.4.

Other methods

The red, green, blue, and depth (RGB-D) image method represents a 3-D scene by composing sectional or layered color 2-D images at different depths.^85–87 Each RGB-D image is a set of color 2-D and depth images. We can obtain RGB-D images using commercially available RGB-D cameras, such as the Microsoft Kinect. A hologram can then be generated from diffraction calculations for the light passing from each sectional color 2-D image to the hologram plane. A 3-D Fourier-spectrum approach^88,89 can yield planar as well as cylindrical holograms.

2.2.5.

Comparison

Table 2 summarizes advantages and disadvantages of three CGH-calculation methods: the point-cloud, polygon, and multiview-image methods. The point-cloud method can be used to reconstruct very-high-definition 3-D images because the complex amplitude on the CGH plane is calculated based on wave optics. However, the computational cost associated with the point-cloud method is very high because computational complexity is proportional to both the number of point-light sources and the number of pixels in the CGH. Hence, it is important to reduce the computational complexity of the point-cloud method by implementing algorithms, such as a recurrence-relation algorithm,^10–13 a patch-model algorithm,^18,19 an LUT algorithm,^20–28 or other algorithms. Like the point-cloud method, the polygon method is based on wave optics and can further be used to reconstruct high-definition 3-D images; however, it is associated with a high computational cost. By contrast, the multiview-image method is based on geometrical optics, and, because various techniques commonly used for 3-D computer graphics based on geometrical optics can be applied to generate 2-D images for different viewpoints, the computational cost is low. Moreover, occlusion culling (or hidden-surface removal), shading, and texture expression can be easily performed in the multiview-image method. However, the resolution of the reconstructed 3-D image is relatively low because the phase information of the light is lost.

Table 2

Advantages and disadvantages of CGH-calculation methods.

Method	Image quality	Computational cost	Ease of occlusion culling, shading, and texture expression
Point-cloud	Very high	Very high	Very complicated
Polygon	High	High	Complicated
Multiview-image	Low, medium	Low	Easy

3.

Real-Time Holographic Aerial-Projection System Based on Image-Type Holograms

In this section, we review a real-time holographic aerial-projection system that can yield sufficiently short computation times to allow real-time calculation of CGHs and manipulation of aerial 3-D images. Image-type holograms²⁹ are generated from a 3-D object that is located on or close to the hologram plane. The computational load required for point-cloud calculations generally increases with the square of $z_l$, which represents the distance between a 3-D object and the hologram plane, as shown in Fig. 5, because the range-dependent calculation of Eq. (1) is limited by the pixel pitch $p$ of the SLM. We can reduce the computational load with image holograms, but they are difficult to project as aerial 3-D images because image-type holograms can only reconstruct images close to the hologram plane. To address this issue, we included two parabolic mirrors to achieve aerial projection of 3-D images. Figure 6(a) shows a schematic of the proposed holographic aerial-projection system. Two parabolic mirrors with holes in their centers and placed face-to-face on top of each other are used. An SLM is positioned at the lower hole of the parabolic mirrors. The SLM is inclined at an angle of $\sim 20\mathrm{deg}$ relative to the horizontal plane such that the diffraction waves from the CGHs are incident on the upper parabolic mirror. An image-type CGH is displayed on the SLM, and a holographic image is reconstructed near the SLM plane. Then, the parabolic mirrors project a real image of the reconstructed holographic image at the upper hole, allowing a view of the holographic image. Figure 6(b) shows an overview of the experimental setup. We used a green laser operated at 532 nm as the optical source. The outer diameter, hole diameter, height, and focal length of each parabolic mirror are 288, 80, 55, and 100 mm, respectively. We used a phase-modulation SLM (Holoeye Photonics AG) with a pixel pitch, resolution, refresh rate, and gradation of $8.0\mu \mathrm{m}\times 8.0\mu \mathrm{m}$, $1920\times 1080\text{pixels}$, 60 Hz, and 256 gray levels, respectively. The SLM was positioned $\sim 20\mathrm{mm}$ away from the bottom of the parabolic mirrors. Figure 7(a) is a schematic of the 3-D object that was used to confirm the system’s functionality, the skeleton of a fish comprising 1498 point-light sources. The 3-D object was virtually placed $\sim 20\mathrm{mm}$ away from the CGH plane. Figures 7(b)–7(d) show images of the projected motion picture captured by a digital video camera. The position and size of the fish were varied using keyboard operations in real time. In Figs. 7(b) and 7(c), the position of the 3-D object was varied along the $y$- and $z$-axes, respectively. Because the focal distance of the camera was fixed, the projected images became out of focus as the 3-D object was moved along the $z$-axis. In Fig. 7(d), the size of the 3-D object was increased. The results demonstrate that real-time holographic aerial projection is possible with image-type CGHs and two parabolic mirrors.

Fig. 5

Range-dependent calculation of Eq. (1).

Fig. 6

Real-time holographic aerial-projection system based on image-type holograms: (a) a schematic and (b) a photograph of the setup.

Fig. 7

Experimental results from (a) a 3-D object comprising 1498 point-light sources and the obtained holographic images using keyboard operations in real time as (b) the 3-D object is moved along the $y$-axis, (c) the 3-D object is moved along the $z$-axis, and (d) the 3-D object is enlarged.

To evaluate real-time performance of the experimental system, we measured the processing time for CGH calculation when the number of the point-light sources and the distance between the 3-D object and the CGH plane were changed. Here, we set the pixel pitch and the resolution of CGHs to be the same as those used in the experiment described above, respectively. We applied Nishitsuji’s algorithm³⁹ to accelerate image-type CGH calculation. We used the following software environment for this system: Microsoft Windows 7 Professional 64bit; 3.5-GHz Intel Core i5-4690 CPU with 32 GB memory (we fully used four cores); Microsoft Visual C++ 2013. Figure 8 shows the measurement results summarizing the relationship between the number of points and the calculation time depending on $z_l$. Here, the processing time for displaying a CGH was small enough to be negligible with respect to the CGH calculation time. For simplicity, we used the same values of $z_l$ for each point-light source in this measurement. In Fig. 8, both the horizontal and vertical axes are described by a common logarithm. The calculation time was the average of 1000 measurements. Figure 8 shows that the CGH calculation time tends to be proportional to the number of point-light sources when the number of point-light sources is more than 100,000 points. By contrast, the acceleration efficiency is low when the number of point-light sources is small. This is caused by the calculation process of Nishitsuji’s algorithm, which consists mainly of the following two steps: generation of zone-plate patterns and synchronization of the patterns. While the calculation time of the former step is independent of the number of point-light sources, the calculation time of the latter step is proportional to the number of point-light sources. This means the calculation time of the latter step is short enough to be negligible when the number of point-light sources is large. Figure 8 also shows that the CGH calculation time is mostly proportional to the square of $z_l$, and we can realize real-time reconstruction of a 3-D object consisting of $\sim \mathrm{100,000}$ point-light sources as $30\text{frames}/\mathrm{s}$ when $z_l=2\mathrm{mm}$.

Fig. 8

Measurement results summarizing the relationship between the number of points and the calculation time depending on $z_l$.

4.

Application in a Holographic Augmented-Reality Display

Next, we review a real-time holographic display that computes CGHs with a GPU performing calculations in parallel. The point-cloud method can be accelerated by processing the calculations of Eq. (3) in parallel since Eq. (3) can be independently calculated for each pixel. Hence, we can realize real-time reconstruction of holographic images by the point-cloud method using a GPU.^5–9 However, even more acceleration of CGH calculation is required for large holographic displays. In addition, the pixel pitch of commercially available SLMs is too coarse to achieve large reconstructed images with large viewing angles. To avoid the issue, head-mounted or near-eye holographic displays have been researched.^86,90–94 A near-eye display does not require large SLMs; the SLMs only need to cover the field of view of a human eye. Since the view point is fixed in the near-eye display, a large viewing angle is not needed. For these two benefits, we chose to demonstrate dynamic augmented reality (AR) with a near-eye display. Figures 9(a) and 9(b) show a schematic of a holographic AR display system and a photograph of its prototype for preliminary experiments, respectively. We constructed the system based on Ichikawa’s Fourier-transform based optical setup.⁹⁵ We used 532-nm green lasers as optical sources. Since the optical path is bilaterally symmetric in Figs. 9(a) and 9(b), we explain only the left side of the setup. Light emitted from the laser is expanded by the microscope objective; is reflected by the first half mirror; is collimated by the collimator lens, whose focal length is $\sim 300\mathrm{mm}$; and illuminates the SLM. The SLM has the same performance as that used in the Sec. 3. Reconstructed light from the SLM passes through the lens and the first half mirror and is reflected by the second half mirror. By observing the reconstructed light, holographic images can be seen behind the second half mirror. Here, we applied the single-sideband method in the display system and half-zone-plate processing for CGH calculations. These methods allow us to reconstruct clear holographic images by filtering out nondiffracted light and the conjugate image with a shading plate.^96,97

Fig. 9

Top view of (a) schematic and (b) photograph of the holographic AR display system for preliminary experiment. MO: microscope objective; HM: half mirror; CL: collimator lens; SP: shading plate; and M: mirror.

First, we confirmed that the constructed system could project a holographic image at variable distances. We positioned real objects 0.4 and 0.6 m away from viewpoint for the left eye and reconstructed a CGH generated by a framework model of a cube’s edges comprising 284 point-light sources. Figure 10 shows images captured by a digital camera positioned at the viewpoint. The value (0.4 m or 0.6 m) of the real object in each figure indicates the distance between the real object and the viewpoint; we focused the digital camera on the real object when capturing the images of Fig. 10. In Figs. 10(a) and 10(b), we projected the CGH of a cube positioned 0.4 m away from the viewpoint. In Figs. 10(c) and 10(d), we reconstructed a CGH of a cube positioned 0.6 m away from the viewpoint. Observe that the reconstructed images of Figs. 10(a) and 10(d) are in focus and those of Figs. 10(b) and 10(c) are out of focus. This test demonstrates the feasibility of projecting holographic images at varying distances for use in AR applications.

Fig. 10

Images projected by the holographic AR display system. (a) and (b) Holographic images are projected 0.4 m away from the viewpoint. (c) and (d) Holographic images are projected 0.6 m away from the viewpoint. Panels (a) and (c) are focused at 0.4 m away from the viewpoint. Panels (b) and (d) are focused at 0.6 m away from the viewpoint.

Next, we introduced a motion sensor into the constructed system to allow a viewer to interactively manipulate holographic images.^9,98–100 Figure 11 shows the processing steps for the interactive holographic AR display system. First, we obtain the initial coordinates of a virtual 3-D object and calculate the CGH from the object as described above. The calculated CGH is displayed with the SLM, and a holographic image of the 3-D object is projected. Next, the system detects the horizontal and depth positions of a viewer’s forefinger using a motion sensor. After detecting the position of the forefinger, the system changes the horizontal and depth coordinates of the artificial object so that the projected holographic image follows the forefinger. Subsequently, the system calculates a hologram of the object using the changed coordinates and displays the CGH on the SLM. These steps in a loop allow interaction with the hologram. We used the same object as in the above experiments with the near-eye display. We used a leap motion sensor (Leap Motion, Inc.) as the motion sensor.¹⁰¹ In the same environment as used in Sec. 3, an NVIDIA Geforce GTX 980 GPU (1279-MHz GPU clock, 3505-MHz memory clock, 4096 MB memory, and 2048 cores) was used to enable real-time calculation of CGHs. A compute unified device architecture (CUDA) version 8.0 was used as an integrated development environment for the GPU. Figure 12 illustrates the experimental results. A motion picture of the projected holographic images was captured by a digital camera at a fixed focal plane; 15 frames were extracted from the motion picture. In Figs. 12(a) and 12(b), the forefinger was moved from left to right and from right to left, respectively. Although the holographic image was observed to have a short latency due to the CGH calculation time, the holographic image followed the movement of the forefinger in real time. In Fig. 12(c), the forefinger moved from front to back along the depth direction. Because the focal distance of the camera was fixed, the holographic image was initially out of focus, gradually approached the in-focus position, then passed the focal distance, and gradually became unfocused once again. Therefore, these results demonstrated the interactive handling of holographic AR images is feasible with the developed apparatus.

Fig. 11

Processing procedure for the interactive holographic AR display system.

Fig. 12

Images of the interactive holographic AR display system when the forefinger was moved (a) from left to right, (b) from right to left, and (c) from front to back.

To characterize the interactive handling performance of the system, the processing time for constructing the holographic AR display system was evaluated. Here, only the CGH calculation time was measured because the processing time for the forefinger detection was less than 0.1 ms and, thus, was considered to be negligible. Figure 13 shows the measurement results as $z_l$ was varied in 100 mm increments in the range of 100–500 mm. The calculation time was constant except for when $z_l=100\mathrm{mm}$. This is because the range-dependent calculation of Eq. (1), as shown in Fig. 5, is satisfied when $z_l$ is larger than 200 mm; the range requiring CGH calculation corresponds to the whole CGH size. Figure 13 also shows that interactive handling can be realized using the constructed system at $30\text{frames}/\mathrm{s}$ when the number of point-light sources is $\sim 2000$. In the program, “if” statements were used to judge whether the calculation of Eq. (1) is necessary or not, which reduces the speeding-up ratio in the GPU implementation. Hence, for comparison, the CGH calculation time was also measured without the use of “if” statements. The results show that the calculation time is $\sim 1.4$ to 2.0 times faster without “if” statements compared to that with “if” statements.

Fig. 13

Measurement results summarizing the relationship between the number of points and the calculation time depending on $z_l$ in the holographic AR display system.

5.

Conclusion

We reviewed two strategies to improve the real-time performance of electroholographic systems that project aerial 3-D images. The first used parabolic mirrors to project holographic images reconstructed from image-type CGHs in the air without increasing the computational burden. It was shown that holographic aerial projections can be realized in real time with image-type CGHs and two parabolic mirrors while the position and size of a 3D fish were varied using keyboard operations. The processing time for the CGH calculation was measured to evaluate the real-time performance of the system. The measurement results indicate that real-time reconstruction of a 3-D object consisting of $\sim \mathrm{100,000}$ point-light sources can be done at $30\text{frames}/\mathrm{s}$ by the proposed apparatus when the distance between the 3-D object and the hologram plane is about 2 mm. The second system parallelizes CGH calculations using a GPU instead of HORN system. A holographic AR display system was constructed based on a near-eye display. The feasibility of projecting holographic images at varying distances was demonstrated for AR applications. In addition, a Leap Motion sensor was implemented into the constructed system as a motion sensor to allow a viewer to manipulate holographic images interactively. Although the holographic image was shown to have only a short latency due to the CGH calculation time of $30\text{frames}/\mathrm{s}$ with 2000 point-light sources, it was demonstrated that the holographic image followed the forefinger in real time and that interactive handling of holographic AR images can be realized with the proposed apparatus. We measured the CGH calculation time of the holographic AR display system. In future studies, we will aim to develop a mobile-sized multimodal and multifunctional 3-D display system by incorporating the HORN system¹⁷ with sensing and haptic devices into the holographic display system.