1.

Introduction

Integral imaging is a kind of autostereoscopic three-dimensional (3-D) display and has been actively studied because it does not require any special viewing aids such as polarization glasses, which is modified from integral photography invented by Lippmann.¹ The static integral photography with the photographic films was improved into the active integral imaging by virtue of the advanced imaging and display devices.² The integral imaging is composed of the two main processes, the pickup and the display processes. In the pickup process, a number of images that have different perspectives are captured by the imaging device such as charge-coupled device (CCD) and complementary metal-oxide semiconductor camera, which are called the elemental images and reconstructed into the integrated 3-D image in the display process.

The elemental image can be generated by several methods. The typical method is the direct pickup method where the elemental images formed by the lens array are directly captured using the imaging device.³ This method can obtain the elemental image from the real object easy and simply, but there are some restrictions such as the limitations of the object size and the location, degradations of the image quality from the resolution limitation of the imaging device and the elemental lens aberration, and the pseudoscopic problem for real image reconstruction. The modified direct pickup methods using the additional lens system have been proposed to resolve the pseudoscopic problem.^4–6 The pickup method without the lens array has been proposed to overcome the image quality degradation due to the limitation of the resolution of the imaging device and the aberration from the lens array, which is performed by the multiple capturing processes using several cameras or a camera on a moving frame.^7,8 The method using the depth map obtained by the depth camera instead of the lens array has also been proposed.⁹

The elemental images are formed due to the simple geometrical principles by the lens or the pinhole array; therefore, the pickup process can be performed by computer calculation and drawing.^10,11 Several algorithms about elemental image generation have been proposed and reported. Multiple viewpoint rendering (MVR) is similar to the multiple capturing process.⁷ Each elemental image of the corresponding elemental lens is captured on the proper scale by the virtual camera sequentially for all elemental lenses in the lens array and quilted together in the proper position. Because MVR is based on the optical structure of the lens array, the perspectives of the elemental images are very accurate. However, MVR is not efficient in view of the redundancy among the elemental images, and the processing amount and time are increased tremendously in proportion to the number of elemental lenses. For more effective and quick processing, modified algorithms such as parallel group rendering (PGR) and viewpoint vector rendering (VVR) have been proposed,^12,13 which reduce the usage count of the capturing process through rearrangement of the pixels of the orthographic views.

In this paper, we propose a pickup method for real object adopting the VVR algorithm. The orthographic views for elemental image generation are captured using the telecentric lens system. The pseudoscopic problem for real image reconstruction can be resolved by adjusting the pickup position and direction. Using the proposed method, the elemental images of the real object can be efficiently generated for the real and virtual display modes without the pseudoscopic problem. The introduction of algorithms is explained in Sec. 2. The details of the proposed system and the experimental results follow. The feasibility of the proposed method is proved by the reconstructed 3-D images from the high-quality elemental images.

2.

Algorithms of Elemental Image Generation

Figure 1 shows the schematic diagram of MVR and PGR. In MVR, as shown in Fig. 1(a), each elemental image for the corresponding elemental lens is captured by a virtual camera whose view angle is adjusted to the $f$-number of the elemental lens. The capturing locations of the virtual camera are the centers of all elemental lenses in the lens array. A set of elemental images is obtained to attach the elemental images on the proper positions. Since the scene captured by the virtual camera can be controlled delicately, the elemental image generated by MVR has a high degree of accuracy.

Fig. 1

Schematic diagrams of pickup algorithms: (a) multiple viewpoint rendering and (b) parallel group rendering.

However, the number of capturing processes in MVR, which is the most time and data consuming process in the generation algorithm, is proportional to that of the elemental lenses. Therefore, MVR needs huge amounts of image data and processing time for a large-size scene and a lens array whose elemental lenses abound. Moreover, the elemental image described for the real display mode using MVR incurs the pseudoscopic problem similarly with the direct pickup process.

PGR is proposed to reduce the processing amount and time compared with MVR. In PGR, as shown in Fig. 1(b), the pictures whose pixel number is the same as the number of elemental lenses are captured for the directions of view within the viewing angle. The number of capturing in PGR, $N_{\mathrm{PGR}}$, is the same as that of pixels in the region of one elemental lens. Therefore, $N_{\mathrm{PGR}}$ is not influenced by the total number of elemental lenses and the size of the reconstructed scene but determined by only the pixel number per the elemental lens region, and can be expressed as

The elemental image generated by PGR, however, is suitable only for the focused display mode, where the gap between the display panel, $g$, and the lens array is set to the focal length of the lens array, $f$, because one elemental lens is represented by one pixel according to the concepts of the PGR algorithm and the focused display mode.¹⁴ Since the integrated 3-D image of the focused display mode can be located in front of and behind the lens array with the same resolution of the lens array, which means the pixel size of the reconstructed image is same as the size of the elemental lens, PGR is suitable for the case when the size of elemental lens is small so the number of pixels for one elemental lens region is few.

VVR is evolved from PGR to generate elemental images for the real and virtual display modes. Figure 2 shows the basic concept of VVR. As shown in Fig. 2(a), the segmental images at the same local position in every elemental image are collected from the orthographic scene which is captured for a specific view direction, named a viewpoint vector.¹⁵ Figure 2(b) shows the orthographic views according to the viewpoint vectors. The number of the view vectors is equal to the capturing number, $N_{\mathrm{VVR}}$, and decided by the magnification factor, $M$, which is calculated by the ratio of $g$ and the location of the central depth plane (CDP), $L$ and described as

Fig. 2

Concept of viewpoint vector rendering (VVR): (a) schematic diagram; (b) orthographic views according to viewing vectors.

The size of the segment, $D_S$ shown in Fig. 2(a), is also determined as the ratio between the size of elemental lens and the number of view vector at once.

We can assume that the PGR is the special case of VVR when the size of segment is the same as the pixel size ($D_S=P_X$). Because there is no meaning for the segment smaller than the pixel, $N_{\mathrm{VVR}}$ cannot be above $N_{\mathrm{PGR}}$. In VVR, since the location of CDP can be controlled in front of and behind the lens array, the elemental images for the real and the virtual display modes can be generated unconstrainedly. Compared with MVR, VVR keeps up the efficient advantages of PGR. In this paper, we propose the pickup process for a real object adopting VVR to generate the elemental image, which can be reconstructed in the real and virtual display modes.

3.

Proposed Pickup Process for Real Object

As mentioned above, the purpose of this paper is the realization of the pickup process for the real world adjusting the advanced pickup algorithm for the virtual space, VVR. To implement the pickup system, the real devices corresponded with the virtual tools must be found and applied while the advantage of VVR such as the compatibility of the display modes and the reduction of the capturing number is preserved. One of the important issues to compose the pickup system using VVR is how to take the orthographic pictures. In this paper, the telecentric lens system is instrumented to make the orthographic scenes, which is the compounded lens system whose entrance or exit pupil is located at infinity and can produce an orthographic view of the objects.

The pseudoscopic problem is one of the original drawbacks of the integral imaging to reconstruct the integrated 3-D image in the real display mode, which is the phenomenon that the location of the reconstructed image is inverted back to front. Figure 3 shows the cause of the pseudoscopic phenomenon. As shown in Fig. 3(a), the direction of pickup is opposite to that of display and the image of the object nearer to the pickup device is located farther from the observer in the real display mode. Therefore, the observer should watch the reconstructed 3-D image backward. In the virtual display mode, the directions of pickup and display are the same and the pseudoscopic problem vanishes.

Fig. 3

Directions of the pickup and the display processes for two display modes: (a) real mode; (b) virtual mode.

The pseudoscopic problem of the elemental image generated for the real display mode can be resolved to rotate each elemental image 180 deg centrosymmetrically. In this case, however, the elemental image for the real display mode is converted into that for the virtual display mode.⁴ In computer graphics, the pseudoscopic objects can be generated easily to make the elemental image for the orthoscopic 3-D image in the real display mode, which is impossible in the real world.

In the pickup process adopting VVR, the pseudoscopic problem can be resolved by adjusting the direction and the position of the imaging device. Figure 4 shows the schematic concept of the pickup process adopting VVR. The principle of the proposed pickup process is very similar to that of the two-step integral imaging in which the reconstructed image of the first step is used as the object of the second step.¹⁶ For the one-step process, multiple cameras placed in front of the CDP should directly capture the object using the image mapping. However, it is unrealizable because the CCD sensor of the general camera, which captures only the perspective images, should be relatively large for the generation of virtual mode of elemental image or even capture the virtual object for the real mode. The one-step process without these limitations can be performed using multiple telecentric lenses. Figures 4(a) and 4(b) show the generation of elemental image for the virtual and real modes, respectively. The directions of pickup and display in both cases are not opposite but identical. Therefore, using the process shown in Fig. 4(b), the orthoscopic elemental image for the real display mode can be generated without the pseudoscopic virtual object, which can be applied to the pickup system for a real object.

Fig. 4

Schematic concept of the pickup process adopting VVR: (a) orthoscopic elemental image for virtual display mode; (b) orthoscopic elemental image for real display mode.

4.

Experimental Results of Pickup and Reconstruction

To verify the proposed method, experiments to generate the orthoscopic elemental images for the real and virtual display modes are performed for the virtual and real objects. The integrated 3-D images using the obtained elemental images are reconstructed to prove the proposed solution of the pseudoscopic problem.

Figure 5 shows the virtual and real 3-D objects used in the VVR pickup experiments. The virtual objects, as shown in Fig. 5(a), are composed of five characters; K, H, U, 3, D. The characters are placed at intervals of 10 mm and the total depth is 40 mm. The real objects shown in Fig. 5(b) are composed of two dice whose side is 5 mm.

Fig. 5

Three-dimensional objects used in experiments: (a) virtual object; (b) real object.

Figure 6 shows the experimental setup for the pickup and the display. We use the telecentric lens and the rotation stage to capture the orthographic pictures in the proper directions. The telecentric lens whose diameter is 156.19 mm and working distance range is about 170 to 220 mm is used for the pickup system. For the display system, the simple integral imaging system which consists of the lens array and retina display is implemented as shown in Fig. 6(b). The specifications of the experimental equipment are shown in Table 1.

Fig. 6

Experimental setup: (a) pickup; (b) display.

Table 1

Specifications of the equipment.

Figure 7 shows the elemental images obtained by the pickup process for the virtual and the real objects. Since the location of CDP is set to 60 mm in all cases, the total number of the viewpoint vectors is 5 by 5 and the number of capturing processes is 25 times. For the case of the MVR pickup system whose specifications are the same, because the number of elemental lens in the active area is 150, the capturing process must be performed 150 times. Therefore, using the proposed method, the elemental image can be efficiently generated from the view point of the number of capturing process. More details of the setup system are shown in Table 2.

Fig. 7

Elemental images of virtual objects (a) for real display mode and (b) for virtual display mode, and real objects (c) for real display mode and (d) for virtual display mode.

Table 2

Specifications of the system factors.

Videos 1(a) and 1(b) show the reconstructed 3-D images for the virtual objects, whereas Videos 1(c) and 1(d) show those for the real objects in the real and virtual display modes, respectively. The viewing angles are 21 deg in the real display mode, which are the case of Videos 1(a) and 1(c), and 23 deg in the virtual display mode, which are the case of Videos 1(b) and 1(d). All reconstructed 3-D images are orthoscopic because the perspectives of the 3-D images are the correct views for the viewing directions as shown in Video 1. Therefore, the proposed method is proved to resolve the pseudoscopic problem in the real display mode.

Video 1

Integrated 3-D images for virtual objects (a) in real display mode (MPEG, 3.2 MB) [URL: http://dx.doi.org/10.1117/1.OE.53.7.073109.1] and (b) in virtual display mode (MPEG, 3.5 MB) http://dx.doi.org/10.1117/1.OE.53.7.073109.2], and for real objects (c) in real display mode (MPEG, 3.0 MB) http://dx.doi.org/10.1117/1.OE.53.7.073109.3] and (d) in virtual display mode (MPEG, 4.8 MB) http://dx.doi.org/10.1117/1.OE.53.7.073109.4].

5.

Discussion and Conclusion

In this paper, we propose the pickup method using VVR, which is the modified algorithm for elemental image generation, and verify the feasibility of the proposed system through the experiments of pickup and reconstruction. The number of capturing processes can be reduced in the proposed system compared with the MVR pickup system, where the size of captured scene must be large enough. Therefore, the high-resolution imaging device must be implemented. The proposed system also has some advantages in comparison with the pickup system using the plenoptic camera, for example, Lytro camera, California. The pickup system using Lytro camera can generate the elemental images by the single-shot process but there are some limits. First, there is the trade-off relation between the image resolution and the depth of field (DOF). Because Lytro camera focuses on the latter for recording the depth information, the image resolution of Lytro camera is highly degraded compared with that of the digital single-lens reflex camera. Second, the field-of-view (FOV) is fixed. Lytro camera does not require the “focusing” process due to the microlenses; however, the gap between the lens array and the photosensor cannot be controlled. In the proposed system, the high-resolution image can be obtained using the relatively low specifications of the CCD with the reasonable capturing numbers. Also, the FOV can be adjusted by changing some system factors such as the rotation angle of the telecentric lens and capturing numbers. Moreover, we will study about the pickup systems for the generation of the orthographic images using the telecentric lens concepts.

In the proposed system, the elemental image is compounded from the segments of the orthographic scene, which is captured by the imaging system using the telecentric lens. Thus, the specifications of the telecentric lens system such as the diameter and the working distance range make the boundaries of the pickup conditions, which are the size of the object, the locations of CDP, and the range of the object depth. Moreover, there are some errors in the orthographic scenes from the practical telecentric lens, which incurs the discontinuity of the elemental image and causes the failure of integration and the noise of the reconstructed image. The high-quality telecentric lens system should be necessary to capture big objects. However, in our experiments, the errors of the pickup system are tolerable and the noise of the reconstructed image is also ignorable, because the pickup angles of the proposed system are relatively small, as shown in Table 2 of Sec. 4. In the VVR pickup system, the pickup angles which cause the errors mentioned above must be smaller than the viewing angle of the integral imaging. Therefore, those errors can be critical if the viewing angle of the system becomes large. Likewise, we can determine that the range of the object depth nearly depends on the DOF of the telecentric lens when the system has the narrow viewing angle.

The VVR pickup system proposed in this paper can be applied to not only the integral imaging system but also the 3-D pickup system where the high-quality elemental image is required, which can be used for the fields of 3-D application such as 3-D recognition and information processing.