Method for generating planar computer-generated hologram at free viewpoint from cylindrical object light

Abstract. Computer-generated holography (CGH) is an ideal three-dimensional (3D) display technology, and head-mounted displays (HMDs) using CGH are expected to become the next-generation display devices without eye strain, which is called holographic HMDs (holo-HMDs). However, the CGH computational load is too much for a holo-HMD, which has limited processing power. We have devised a method that calculates the cylindrical object light on a sender, broadcasts the object light data to multiple holo-HMDs, and generates holograms matched to the planar display device on the holo-HMD. The cylindrical object light calculations, which take up most of the computational load, are performed on a high-performance computer with sufficient computational power. The cylindrical object light is transformed into an object light on the plane in accordance with the motion of the holo-HMD. The phase difference information between the complex amplitude distribution of the cylindrical object light and the object light on the plane corresponding to a free viewpoint position must be corrected. Using this method, objects placed near the center of the cylinder can be observed from a wide area, making it possible to support 3D image observation with multiple holo-HMDs. Experiments using an actual optical system demonstrated that planar transform was performed and that the computational load on a holo-HMD side was very small.


Introduction
With the recent development of three-dimensional (3D) display technology, the practical application of virtual reality using a head-mounted display (HMD) has advanced. However, when the wearer looks at an object, physiological factors such as vergence can cause discomfort and eye strain, commonly referred to as "3D sickness." Next-generation HMDs are expected to use holography (holo-HMD), which is considered to be an ideal stereoscopic display technology that satisfies human visual characteristics without 3D sickness. [1][2][3][4] It is a key device of ultrarealistic communication technology, 5,6 which aims to provide the sensation of being in the same place at a distance using stereoscopic images.
The holo-HMD requires the hologram data that is generated from virtual objects by a computer, which is called a computer-generated hologram (CGH). 7 However, the calculation of hologram data requires a large amount of computing time and resources such as computation power and memory because holograms have a large number of pixels. Especially, the hologram data take a lot of calculation time to generate. Various acceleration algorithms have been studied to solve the problem, but real-time calculations are not possible with the computer resources of a personal computer (PC) or an HMD. Therefore, real-time calculations have been studied using a large computer cluster or specially designed system for hologram-data calculation, which have huge computational power. [8][9][10] This paper discusses a broadcast model for multiple users (multiple holo-HMDs), in which a high-performance computer calculates data that require a large amount of computing time and resources. It is expected to be used in applications where multiple users observe 3D images simultaneously. This technology is expected to be used in various fields such as medicine, education, and entertainment.
We propose a multi-user system by broadcasting cylindrical object light data, and algorithms to generate hologram data for each user at free viewpoints from cylindrical object light data.

Broadcast Model Using Cylindrical Object Light
The holo-HMD consists of a display device that displays hologram data, a light source (reconstruction light), and an optical system such as lenses and prisms. Hologram data are an interference fringe pattern obtained by interfering with a reference light and a light wave emitted from an object (object light). In CGH, object lights take a large amount of calculation time and resources to calculate. Figure 1 shows a schematic diagram of the broadcast model, in which a high-performance computer such as a computer cluster and a supercomputer calculates object light data and broadcasts it to multiple users. [11][12][13] The use of a high-performance computer enables real-time calculation of object light. Since hologram data depend on the structure of the optical system and specifications of display device such as a pixel pitch and a panel size, the hologram data need to be suitable for each holo-HMD device. To calculate the hologram data, the same object light can be used regardless of the device, then each device only needs the calculation interfering with a reference light and object lights in accordance with the specifications. The calculation from the object light to the hologram data is simple and incurs low processing load (see Sec. 3). Therefore, the processing performed at the receiver side must be computationally less demanding, and the computational cost must also be kept small in the process. PCs and HMDs with a poor computational power are able to generate hologram data in real time and use at various locations, including the home and remote locations.

Cylindrical Object Light
Viewing the 3D image from a free position on the basis of the user's movement provides higher experience to the user. One method of free viewpoint observation is an on demand in which the broadcasting side sends hologram data in accordance with each point of view of the user's HMD. However, this method is inefficient in terms of calculation and communication.
When a user observes an object at 360-deg horizontally, the viewpoint will move along a cylindrical surface surrounding the object. Therefore, we propose a system in which a sender broadcasts light wave data on cylindrical object light surface (COLS), and each holo-HMD device generates the hologram data. It is very easy for a holo-HMD to calculate the hologram data because of the object light on the COLS is calculated by the sender in advance. Furthermore, this system enables multiple users to view 3D images at each free viewpoint using the same object light.

Calculation of Object Light on Cylindrical Surface
Conventional methods for calculating object lights on the cylindrical surface are mainly based on the Fourier transform because of the fast calculations. 14 However, in these methods, objects cannot be defined with a high degree of freedom because the shape definition of an object is limited such as cylindrical objects and plane objects. On the other hand, the point light method, in which an object is defined by a set of point lights, enables more complex objects to be defined. 15 Although in this method, a hologram is difficult to generate in real time on a PC or an HMD, we use the method due to more powerful computers being used in the future broadcast model. Figure 2 shows the coordinates of object-light calculation on COLS. The light wave from each point light with the complex amplitude of A i is propagated to the COLS. The light wave O c ðx; y; zÞ is given as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 1 1 6 ; 3 5 4 O c ðx; y; zÞ ¼ E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 1 1 6 ; 2 9 2 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 1 1 6 ; 2 5 9 where λ is wave length, N is the number of the point lights, ϕ i is a random phase, and r i is the distance between the point light ðx i ; y i ; z i Þ and a point on the COLS.

Transform Cylindrical Object Light into Planer Recording Surface
The display device such as a liquid crystal display and a digital mirror device used in a holo-HMD cannot use cylindrical object light directly because of its planer shape. Therefore, the object light must be transformed into an object light on a planer surface. We assume a tangent plane touched the COLS, on which the transformed object light is recorded [planar recording surface (PRS)]. The positional relationship between the COLS and the PRS is shown in Fig. 3. In the figure, the PRS is centered on the z-axis, but it can be placed at any position along the COLS. The line through a point p i on the PRS and the center of the cylinder is intercepted with the COLS at the point C r ðx; y; zÞ. If the optical path difference Δl between C r and p i is sufficiently small and objects are located near the center of the COLS, the light wave O p ðx p ; y p ; rÞ at the p i is approximately propagated from the C r with an optical path difference Δl. The assumption is defined as (4) This equation shows that the O p is the light wave multiplied the object light sampled at the C r and the phase difference of Δl. We call it a "planar transform." Holograms are obtained by interfering this light wave O p with the reference light specialized for the optical system and the display device of each HMD. The resources required to transform to a PRS and calculate the interference are proportional to the number of the pixels of the display device, which is small enough to calculate on a PC or an HMD. Thus, the PRS is the same as the hologram plane.

Limitation of Transform Range
The spatial frequency of the object light on the PRS becomes larger than that on the COLS toward either end of the PRS. When the spatial frequency contained in the object light exceeds the spatial sampling frequency of the display device, it generates noise folding noise, which degrades the holographic reconstructed image. This is called the sampling theorem, and the area to be transformed into a PRS must be limited so as not to deviate from the sampling frequency range. Let the cylindrical radius be r, the height of the PRS be h, and the distance from the origin to the point p i be L. The distance between p i and p iþ1 is p, which is the pixel pitch of the display device (see Fig. 3). The phase difference between pixels is expressed using the directional derivative of h E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 5 ; 1 1 6 ; 2 4 6 Since h is much smaller than r, an approximation to this equation is E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 1 1 6 ; 1 9 2 h ffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi If the wavelength of the light source is λ, the length of the recording plane that can be transformed into a plane on the basis of the sampling theorem is given as It is worthy to note that the planar transform should be done in the range of h in the x, y directions that satisfies the above conditional Eq. (8).

Six Degrees of Freedom
The cylindrical object light at an arbitrary point can be transformed into a planar shape using the planar transform method described above. Considering actual usage scenarios using holo-HMDs, the planar transform must be performed not only moving along the cylinder but also for more complex movements. There are six degrees of freedom (6-DOF) of movement available to a user wearing an HMD in Fig. 4, and all of them must be supported in order for the image to be observed from a free viewpoint For vertical motion of the z-axis direction "up-down" and left-right tilt motion of the x-axis direction "roll" in Fig. 4, no phase correction is required because it can be achieved by only resampling the object light on the COLS. The transform of vertical motion of the z-axis direction up-down and left-right tilt motion of the x-axis direction roll in Fig. 4 are performed by changing the position and direction of the PRS, and to calculate the same processing of the planer transform.
Let the coordinates of the PRS before movement be x and y for horizontal "forward-back" motion in Fig. 4, and the coordinates after the movement be x trans and y trans . The transform of the object light can be expressed using movement width d and cylindrical radius r: 16 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 9 ; 1 1 6 ; 4 9 5 ðx trans ; y trans Þ ¼ r r þ d ðx; yÞ: The coordinates ðx trans ; y trans Þ after this movement are set as the new PRS, and re-sampling the object light from the original PRS to the new PRS in accordance with Eq. (9). Rotation "yaw and pitch" in Fig. 4 can be also achieved by setting these coordinates as the new PRS and performing phase correction calculations in the same way.
As shown in Fig. 5, for the motion of swinging the head left and right on the z-axis direction yaw in Fig. 4, the coordinates of the PRS after rotation, x rot y and z rot y , are expressed using rotation angle θ rot y and cylindrical radius r: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 0 ; 1 1 6 ; 3 6 7 x rot_y z rot_y ¼ x cos θ rot_y x sin θ rot_y þ r : As shown in Fig. 6, for forward and backward tilting motion on the y-axis direction "pitch" in Fig. 4, the new PRS coordinates, y rot p and z rot p , are expressed using rotation angle θ rot p and cylindrical radius r: For the object light transform of yaw or pitch, after the re-sampling the object light from the original PRS to the new PRS in accordance with Eq. (10) or Eq. (11), the object light is achieved phase corrections of optical pass difference Δl.
The parallel shift "left and right" in Fig. 4 can be achieved using a combination of the movements of the yaw and the forward-back. Figure 7 shows an example of each movement in a parallel shift.
First, the PRS is created at the same position of the target plane in the x-axis direction by the planer transform. In the figure, the PRS is tilted −θ rot_p . Second, the new PRS P M with the parallel to the target plane is made from the PRS by the movement of Yaw. In the figure, rotation of θ rot_p means the PRS has the same viewpoint angle as the target plane. Finally, the P M is moved to at the same z-axis position as the target plane by the movement of the forward-back.

Experiments
To certify that the proposed method transforms the cylindrical object light into the object light on PRSs in accordance with a free position, we conducted image reconstruction experiments using an optical system.  Fig. 6 Transform of pitch.
The optical system uses a liquid crystal on silicon (HOLOEYE Photonics AG: Pluto) for displaying hologram data and a 4f optical system for eliminating unwanted lights of zerothorder light and high-order diffractions. The parameters are shown in Table 1. Amplitude-type hologram data are obtained by interfering a plane light as a reference light with the object light on the hologram plane.
A photograph of the optical system is shown in Fig. 8, and a schematic of the virtual object is shown in Fig. 9. The virtual object is a wireframe pentagram, whose wires are composed of 428 point lights.

Planar Transform
In the first experiment, we reconstructed holograms using planar transform on cylindrical object light. Assuming that the user was observing an object placed near the center of the cylinder from an arbitrary angle, we generated planes at three locations, (0 deg, 45 deg, and 90 deg), starting from the x-axis, and used them for the experiment. The reconstructed images and CG image of the virtual object at the same angle are shown in Fig. 10.
These images demonstrate that the proposed method can be used to observe images on a planar display device without losing the advantages of the cylindrical object light.  Fig. 7 Transform of parallel shift.

Horizontal Backward and Forward Motion
In the next experiment, we reconstructed a hologram with horizontal backward and forward motion. Holograms are generated at 3, 6, 9, and 15 cm backward and forward from the COLS. The reconstructed images are shown in Fig. 11. A positive value for the separation distance indicates movement away from the cylinder, and a negative value indicates movement into the cylinder. They were not affected much by movements of 3, 6, and 9 cm, but, for movements of 15 cm, the image is difficult to display correctly. Therefore, if the planes are more than a certain distance apart, a new cylindrical object light must be calculated and the phase must be corrected.

Yaw and Pitch
In the third experiment, we reconstructed holograms with left and right head movements on the z-axis direction yaw and forward and backward tilting movements on the y-axis direction (pitch). Considering the cylindrical radius, the size of the object, and the size of the recording plane, we expected that the reconstructed images would be beyond the viewpoint after a few degrees of rotation processing for both yaw and pitch. Therefore, we checked two rotation angles: 1 deg and 2 deg. As can be seen from the images, the object position changed due to rotation, demonstrating that the object light was appropriately transformed by phase correction for rotation as well.

Vertical Motion and Roll
Vertical motion up-down along the z-axis and left-right tilt motion roll along the x-axis can be achieved by generating in advance the cylindrical object light for the corresponding part on the plane after movement. For vertical motion, holograms were located at the position moved by 50% of the width of the PRS. For roll motion, the planes were generated with rotation angles of 22.5 deg, 45 deg, and 67.5 deg. The actual images reconstructed by the optical system are shown in Figs. 13 and 14. They show that the position of the object changed as specified.

Left and Right Parallel Shift
In the final experiment, we reconstructed holograms with left and right parallel shift. A movement widths, 7.68 mm (50% of horizontal direction of PRS), was assumed, and composite transforms were performed. The reconstructed images are shown in Fig. 15. The reconstructed image   shows the distortion in the shape of the pentagram. This is attributed to the accumulation of errors due to the use of multiple phase correction calculations based on approximations. Therefore, transform algorithms for left and right transform that do not use approximation need to be devised.

Calculation Time
We measured and compared the computation time of the planar transform processing part of the proposed method with that of CGH generation using planar object light without transform. Both holograms are the same size as the planer hologram in Table 1. The computation times are plotted in Fig. 16. In the conventional method that calculates object light and interference on the hologram plane by one computer, the total computation times were 10 to 50 s for every 100 point lights. In the planer transform, the computation time for the plane transform process is constant at about 0.12 s, which is 1/300 of the computation time of the conventional method. This is because the computational complexity of the proposed planar transform process is determined by the number of pixels in the recording plane, and therefore the load on the holo-HMD side where the proposed method is performed is constant. As shown in Fig. 17, the rest of the computation time, except for the cylindrical object light calculation, is occupied by the plane transform process, confirming that the time required for reference light calculation and fringe calculation is short. The average computation time for the cylindrical object light was~52 s (five trials). The computational cost of the proposed method is smaller than that of the cylindrical object light calculation process. In addition, the computation time for each transform process was measured. The computation time is the average of the time taken in each of the aforementioned experiments. Furthermore, the vertical motion is excluded because it is corrected in the same way as the planar transform.
As shown in Fig. 18, there were no substantial differences in the computation times demonstrating that calculation with the proposed method incurs lower processing cost than cylindrical object light calculation.

Conclusion
We have discussed a broadcast model in which the cylindrical object light calculation, which accounts for most of the computation time in CGH, is performed on a high-performance computer, and holograms are generated on a holo-HMDs. We have proposed a transform method for displaying a cylindrical object light on a planar display device of a holo-HMD. The experiments demonstrated that holograms on a planar display device can display reconstructed images using the proposed methods without losing the advantage of cylindrical object light. They also demonstrated that the proposed transform method requires lower processing power than object light calculation, which has a large computational cost.
In this study, we have achieved principle demonstrations of the proposed method, but in the future, the method will need to be applied to an actual multiple holo-HMD system. This study enables CGH to move closer to practical use in terms of computation time. However, the image quality is not sufficient, so future work will include research into the speckle noise reduction and realistic rendering.