31 July 2014 Real object pickup method for real and virtual modes of integral imaging
Author Affiliations +
Optical Engineering, 53(7), 073109 (2014). doi:10.1117/1.OE.53.7.073109
Abstract
Pickup method adopting the modified algorithm to generate the elemental image from virtual objects is proposed to obtain the elemental image for real objects. In the proposed method, the number of capturing processes is reduced compared with the conventional multiple capturing method. The pseudoscopic image problem can be resolved by controlling the position and the direction of the imaging device in the proposed pickup system. The telecentric lens system is used to capture the orthographic scenes, which are divided and compounded into the elemental image. The validity of the proposal is proved by the experimental results of the pickup and the reconstruction.
Yim, Kim, and Min: Real object pickup method for real and virtual modes of integral imaging

1.

Introduction

Integral imaging is a kind of autostereoscopic three-dimensional (3-D) display and has been actively studied because it does not require any special viewing aids such as polarization glasses, which is modified from integral photography invented by Lippmann.1 The static integral photography with the photographic films was improved into the active integral imaging by virtue of the advanced imaging and display devices.2 The integral imaging is composed of the two main processes, the pickup and the display processes. In the pickup process, a number of images that have different perspectives are captured by the imaging device such as charge-coupled device (CCD) and complementary metal-oxide semiconductor camera, which are called the elemental images and reconstructed into the integrated 3-D image in the display process.

The elemental image can be generated by several methods. The typical method is the direct pickup method where the elemental images formed by the lens array are directly captured using the imaging device.3 This method can obtain the elemental image from the real object easy and simply, but there are some restrictions such as the limitations of the object size and the location, degradations of the image quality from the resolution limitation of the imaging device and the elemental lens aberration, and the pseudoscopic problem for real image reconstruction. The modified direct pickup methods using the additional lens system have been proposed to resolve the pseudoscopic problem.45.6 The pickup method without the lens array has been proposed to overcome the image quality degradation due to the limitation of the resolution of the imaging device and the aberration from the lens array, which is performed by the multiple capturing processes using several cameras or a camera on a moving frame.7,8 The method using the depth map obtained by the depth camera instead of the lens array has also been proposed.9

The elemental images are formed due to the simple geometrical principles by the lens or the pinhole array; therefore, the pickup process can be performed by computer calculation and drawing.10,11 Several algorithms about elemental image generation have been proposed and reported. Multiple viewpoint rendering (MVR) is similar to the multiple capturing process.7 Each elemental image of the corresponding elemental lens is captured on the proper scale by the virtual camera sequentially for all elemental lenses in the lens array and quilted together in the proper position. Because MVR is based on the optical structure of the lens array, the perspectives of the elemental images are very accurate. However, MVR is not efficient in view of the redundancy among the elemental images, and the processing amount and time are increased tremendously in proportion to the number of elemental lenses. For more effective and quick processing, modified algorithms such as parallel group rendering (PGR) and viewpoint vector rendering (VVR) have been proposed,12,13 which reduce the usage count of the capturing process through rearrangement of the pixels of the orthographic views.

In this paper, we propose a pickup method for real object adopting the VVR algorithm. The orthographic views for elemental image generation are captured using the telecentric lens system. The pseudoscopic problem for real image reconstruction can be resolved by adjusting the pickup position and direction. Using the proposed method, the elemental images of the real object can be efficiently generated for the real and virtual display modes without the pseudoscopic problem. The introduction of algorithms is explained in Sec. 2. The details of the proposed system and the experimental results follow. The feasibility of the proposed method is proved by the reconstructed 3-D images from the high-quality elemental images.

2.

Algorithms of Elemental Image Generation

Figure 1 shows the schematic diagram of MVR and PGR. In MVR, as shown in Fig. 1(a), each elemental image for the corresponding elemental lens is captured by a virtual camera whose view angle is adjusted to the f-number of the elemental lens. The capturing locations of the virtual camera are the centers of all elemental lenses in the lens array. A set of elemental images is obtained to attach the elemental images on the proper positions. Since the scene captured by the virtual camera can be controlled delicately, the elemental image generated by MVR has a high degree of accuracy.

Fig. 1

Schematic diagrams of pickup algorithms: (a) multiple viewpoint rendering and (b) parallel group rendering.

OE_53_7_073109_f001.png

However, the number of capturing processes in MVR, which is the most time and data consuming process in the generation algorithm, is proportional to that of the elemental lenses. Therefore, MVR needs huge amounts of image data and processing time for a large-size scene and a lens array whose elemental lenses abound. Moreover, the elemental image described for the real display mode using MVR incurs the pseudoscopic problem similarly with the direct pickup process.

PGR is proposed to reduce the processing amount and time compared with MVR. In PGR, as shown in Fig. 1(b), the pictures whose pixel number is the same as the number of elemental lenses are captured for the directions of view within the viewing angle. The number of capturing in PGR, NPGR, is the same as that of pixels in the region of one elemental lens. Therefore, NPGR is not influenced by the total number of elemental lenses and the size of the reconstructed scene but determined by only the pixel number per the elemental lens region, and can be expressed as

(1)

NPGR=(PLPX)2,
where PX is the pixel size of the display panel and PL is the pitch of the elemental lens.

The elemental image generated by PGR, however, is suitable only for the focused display mode, where the gap between the display panel, g, and the lens array is set to the focal length of the lens array, f, because one elemental lens is represented by one pixel according to the concepts of the PGR algorithm and the focused display mode.14 Since the integrated 3-D image of the focused display mode can be located in front of and behind the lens array with the same resolution of the lens array, which means the pixel size of the reconstructed image is same as the size of the elemental lens, PGR is suitable for the case when the size of elemental lens is small so the number of pixels for one elemental lens region is few.

VVR is evolved from PGR to generate elemental images for the real and virtual display modes. Figure 2 shows the basic concept of VVR. As shown in Fig. 2(a), the segmental images at the same local position in every elemental image are collected from the orthographic scene which is captured for a specific view direction, named a viewpoint vector.15 Figure 2(b) shows the orthographic views according to the viewpoint vectors. The number of the view vectors is equal to the capturing number, NVVR, and decided by the magnification factor, M, which is calculated by the ratio of g and the location of the central depth plane (CDP), L and described as

(2)

NVVR=M2=(Lg)2(whenDSPX).

Fig. 2

Concept of viewpoint vector rendering (VVR): (a) schematic diagram; (b) orthographic views according to viewing vectors.

OE_53_7_073109_f002.png

The size of the segment, DS shown in Fig. 2(a), is also determined as the ratio between the size of elemental lens and the number of view vector at once.

We can assume that the PGR is the special case of VVR when the size of segment is the same as the pixel size (DS=PX). Because there is no meaning for the segment smaller than the pixel, NVVR cannot be above NPGR. In VVR, since the location of CDP can be controlled in front of and behind the lens array, the elemental images for the real and the virtual display modes can be generated unconstrainedly. Compared with MVR, VVR keeps up the efficient advantages of PGR. In this paper, we propose the pickup process for a real object adopting VVR to generate the elemental image, which can be reconstructed in the real and virtual display modes.

3.

Proposed Pickup Process for Real Object

As mentioned above, the purpose of this paper is the realization of the pickup process for the real world adjusting the advanced pickup algorithm for the virtual space, VVR. To implement the pickup system, the real devices corresponded with the virtual tools must be found and applied while the advantage of VVR such as the compatibility of the display modes and the reduction of the capturing number is preserved. One of the important issues to compose the pickup system using VVR is how to take the orthographic pictures. In this paper, the telecentric lens system is instrumented to make the orthographic scenes, which is the compounded lens system whose entrance or exit pupil is located at infinity and can produce an orthographic view of the objects.

The pseudoscopic problem is one of the original drawbacks of the integral imaging to reconstruct the integrated 3-D image in the real display mode, which is the phenomenon that the location of the reconstructed image is inverted back to front. Figure 3 shows the cause of the pseudoscopic phenomenon. As shown in Fig. 3(a), the direction of pickup is opposite to that of display and the image of the object nearer to the pickup device is located farther from the observer in the real display mode. Therefore, the observer should watch the reconstructed 3-D image backward. In the virtual display mode, the directions of pickup and display are the same and the pseudoscopic problem vanishes.

Fig. 3

Directions of the pickup and the display processes for two display modes: (a) real mode; (b) virtual mode.

OE_53_7_073109_f003.png

The pseudoscopic problem of the elemental image generated for the real display mode can be resolved to rotate each elemental image 180 deg centrosymmetrically. In this case, however, the elemental image for the real display mode is converted into that for the virtual display mode.4 In computer graphics, the pseudoscopic objects can be generated easily to make the elemental image for the orthoscopic 3-D image in the real display mode, which is impossible in the real world.

In the pickup process adopting VVR, the pseudoscopic problem can be resolved by adjusting the direction and the position of the imaging device. Figure 4 shows the schematic concept of the pickup process adopting VVR. The principle of the proposed pickup process is very similar to that of the two-step integral imaging in which the reconstructed image of the first step is used as the object of the second step.16 For the one-step process, multiple cameras placed in front of the CDP should directly capture the object using the image mapping. However, it is unrealizable because the CCD sensor of the general camera, which captures only the perspective images, should be relatively large for the generation of virtual mode of elemental image or even capture the virtual object for the real mode. The one-step process without these limitations can be performed using multiple telecentric lenses. Figures 4(a) and 4(b) show the generation of elemental image for the virtual and real modes, respectively. The directions of pickup and display in both cases are not opposite but identical. Therefore, using the process shown in Fig. 4(b), the orthoscopic elemental image for the real display mode can be generated without the pseudoscopic virtual object, which can be applied to the pickup system for a real object.

Fig. 4

Schematic concept of the pickup process adopting VVR: (a) orthoscopic elemental image for virtual display mode; (b) orthoscopic elemental image for real display mode.

OE_53_7_073109_f004.png

4.

Experimental Results of Pickup and Reconstruction

To verify the proposed method, experiments to generate the orthoscopic elemental images for the real and virtual display modes are performed for the virtual and real objects. The integrated 3-D images using the obtained elemental images are reconstructed to prove the proposed solution of the pseudoscopic problem.

Figure 5 shows the virtual and real 3-D objects used in the VVR pickup experiments. The virtual objects, as shown in Fig. 5(a), are composed of five characters; K, H, U, 3, D. The characters are placed at intervals of 10 mm and the total depth is 40 mm. The real objects shown in Fig. 5(b) are composed of two dice whose side is 5 mm.

Fig. 5

Three-dimensional objects used in experiments: (a) virtual object; (b) real object.

OE_53_7_073109_f005.png

Figure 6 shows the experimental setup for the pickup and the display. We use the telecentric lens and the rotation stage to capture the orthographic pictures in the proper directions. The telecentric lens whose diameter is 156.19 mm and working distance range is about 170 to 220 mm is used for the pickup system. For the display system, the simple integral imaging system which consists of the lens array and retina display is implemented as shown in Fig. 6(b). The specifications of the experimental equipment are shown in Table 1.

Fig. 6

Experimental setup: (a) pickup; (b) display.

OE_53_7_073109_f006.png

Table 1

Specifications of the equipment.

EquipmentSpecifications
Telecentric lensDiameter (mm)156.19
Working distance (mm)170 to 220
Charge-coupled deviceNumber of pixel H×V3840×2748
Pixel size, H×V (μm)1.67×1.67
Diagonal of pickup device (inch)0.5
Lens arrayNumber of elemental lens H×V15×15
(Active area H×V)15×10
Pitch of elemental lens (mm)5
Focal length (mm)10
Display panelNumber of pixel H×V1136×640
Pixel size, H×V (μm)77.6×77.6

Figure 7 shows the elemental images obtained by the pickup process for the virtual and the real objects. Since the location of CDP is set to 60 mm in all cases, the total number of the viewpoint vectors is 5 by 5 and the number of capturing processes is 25 times. For the case of the MVR pickup system whose specifications are the same, because the number of elemental lens in the active area is 150, the capturing process must be performed 150 times. Therefore, using the proposed method, the elemental image can be efficiently generated from the view point of the number of capturing process. More details of the setup system are shown in Table 2.

Fig. 7

Elemental images of virtual objects (a) for real display mode and (b) for virtual display mode, and real objects (c) for real display mode and (d) for virtual display mode.

OE_53_7_073109_f007.png

Table 2

Specifications of the system factors.

FactorsSpecifications
Rotation angle of the telecentric lensReal mode (deg)−9.5−4.804.89.5
Virtual mode (deg)−14.0−7.107.114
Size of the captured sceneH×V (mm)80×60

Videos 1(a) and 1(b) show the reconstructed 3-D images for the virtual objects, whereas Videos 1(c) and 1(d) show those for the real objects in the real and virtual display modes, respectively. The viewing angles are 21 deg in the real display mode, which are the case of Videos 1(a) and 1(c), and 23 deg in the virtual display mode, which are the case of Videos 1(b) and 1(d). All reconstructed 3-D images are orthoscopic because the perspectives of the 3-D images are the correct views for the viewing directions as shown in Video 1. Therefore, the proposed method is proved to resolve the pseudoscopic problem in the real display mode.

Video 1

Integrated 3-D images for virtual objects (a) in real display mode (MPEG, 3.2 MB) [URL: http://dx.doi.org/10.1117/1.OE.53.7.073109.1] and (b) in virtual display mode (MPEG, 3.5 MB) http://dx.doi.org/10.1117/1.OE.53.7.073109.2], and for real objects (c) in real display mode (MPEG, 3.0 MB) http://dx.doi.org/10.1117/1.OE.53.7.073109.3] and (d) in virtual display mode (MPEG, 4.8 MB) http://dx.doi.org/10.1117/1.OE.53.7.073109.4].

OE_53_7_073109_f008.png

5.

Discussion and Conclusion

In this paper, we propose the pickup method using VVR, which is the modified algorithm for elemental image generation, and verify the feasibility of the proposed system through the experiments of pickup and reconstruction. The number of capturing processes can be reduced in the proposed system compared with the MVR pickup system, where the size of captured scene must be large enough. Therefore, the high-resolution imaging device must be implemented. The proposed system also has some advantages in comparison with the pickup system using the plenoptic camera, for example, Lytro camera, California. The pickup system using Lytro camera can generate the elemental images by the single-shot process but there are some limits. First, there is the trade-off relation between the image resolution and the depth of field (DOF). Because Lytro camera focuses on the latter for recording the depth information, the image resolution of Lytro camera is highly degraded compared with that of the digital single-lens reflex camera. Second, the field-of-view (FOV) is fixed. Lytro camera does not require the “focusing” process due to the microlenses; however, the gap between the lens array and the photosensor cannot be controlled. In the proposed system, the high-resolution image can be obtained using the relatively low specifications of the CCD with the reasonable capturing numbers. Also, the FOV can be adjusted by changing some system factors such as the rotation angle of the telecentric lens and capturing numbers. Moreover, we will study about the pickup systems for the generation of the orthographic images using the telecentric lens concepts.

In the proposed system, the elemental image is compounded from the segments of the orthographic scene, which is captured by the imaging system using the telecentric lens. Thus, the specifications of the telecentric lens system such as the diameter and the working distance range make the boundaries of the pickup conditions, which are the size of the object, the locations of CDP, and the range of the object depth. Moreover, there are some errors in the orthographic scenes from the practical telecentric lens, which incurs the discontinuity of the elemental image and causes the failure of integration and the noise of the reconstructed image. The high-quality telecentric lens system should be necessary to capture big objects. However, in our experiments, the errors of the pickup system are tolerable and the noise of the reconstructed image is also ignorable, because the pickup angles of the proposed system are relatively small, as shown in Table 2 of Sec. 4. In the VVR pickup system, the pickup angles which cause the errors mentioned above must be smaller than the viewing angle of the integral imaging. Therefore, those errors can be critical if the viewing angle of the system becomes large. Likewise, we can determine that the range of the object depth nearly depends on the DOF of the telecentric lens when the system has the narrow viewing angle.

The VVR pickup system proposed in this paper can be applied to not only the integral imaging system but also the 3-D pickup system where the high-quality elemental image is required, which can be used for the fields of 3-D application such as 3-D recognition and information processing.

Acknowledgments

This research was supported by Ministry of Knowledge Economy (MKE), Korea, as a project, “Development of interactive user interface based 3-D system”.

References

1. 

G. Lippmann, “Épreuves réversibles: photographies intégrals,” C. R. Acad. Sci. 146, 446–451 (1908).CASMEI0764-4442Google Scholar

2. 

F. Okanoet al., “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36(7), 1598–1603 (1997).APOPAI0003-6935http://dx.doi.org/10.1364/AO.36.001598Google Scholar

3. 

J.-H. Parket al., “Viewing-angle-enhanced integral imaging by elemental image resizing and elemental lens switching,” Appl. Opt. 41(32), 6875–6883 (2002).APOPAI0003-6935http://dx.doi.org/10.1364/AO.41.006875Google Scholar

4. 

J. Araiet al., “Gradient-index lens-array method based on real-time integral photography for three-dimensional images,” Appl. Opt. 37(11), 2034–2045 (1998).APOPAI0003-6935http://dx.doi.org/10.1364/AO.37.002034Google Scholar

5. 

J.-S. JangB. Javidi, “Formation of orthoscopic three-dimensional real images in direct pickup one-step integral imaging,” Opt. Eng. 42(7), 1869–1870 (2003).OPEGAR0091-3286http://dx.doi.org/10.1117/1.1584054Google Scholar

6. 

J. Hahnet al., “Undistorted pickup method of both virtual and real objects for integral imaging,” Opt. Express 16(18), 13969–13978 (2008).OPEXFF1094-4087http://dx.doi.org/10.1364/OE.16.013969Google Scholar

7. 

M. Halle, “Multiple viewpoint rendering,” in Proc. SIGGRAPH, pp. 243–254, ACM Press, New York (1998).Google Scholar

8. 

Y. Taguchiet al., “TransCAIP: A live 3D TV system using a camera array and an integral photography display with interactive control of viewing parameters,” IEEE Trans. Visualization Comput. Graphics 15(5), 841–852 (2009).TVCG1077-2626http://dx.doi.org/10.1109/TVCG.2009.30Google Scholar

9. 

G. Liet al., “Simplified integral imaging pickup method for real objects using a depth camera,” J. Opt. Soc. Korea 16(4), 381–385 (2012).1226-4776http://dx.doi.org/10.3807/JOSK.2012.16.4.381Google Scholar

10. 

Y. IgarashiH. MurataM. Ueda, “3-D display system using a computer generated integral photograph,” Jpn. J. Appl. Phys. 17(9), 1683–1684 (1978).JJAPA50021-4922http://dx.doi.org/10.1143/JJAP.17.1683Google Scholar

11. 

S.-W. Minet al., “Three-dimensional display system based on computer-generated integral photography,” Proc. SPIE 4297, 187–195 (2001).PSISDG0277-786Xhttp://dx.doi.org/10.1117/12.430817Google Scholar

12. 

R. YangX. HuangS. Chen, “Efficient rendering of integral images,” in Proc. SIGGRAPH, ACM Press, New York (2005).Google Scholar

13. 

S.-W. Minet al., “Enhanced image mapping algorithm for computer-generated integral imaging system,” Jpn. J. Appl. Phys. 45(24–28), L744–L747 (2006).JJAPA50021-4922http://dx.doi.org/10.1143/JJAP.45.L744Google Scholar

14. 

S.-W. MinJ. KimB. Lee, “New characteristic equation of three-dimensional integral imaging system and its applications,” Jpn. J. Appl. Phys. 44(1–7), L71–L74 (2005).JJAPA50021-4922http://dx.doi.org/10.1143/JJAP.44.L71Google Scholar

15. 

Y.-T. Limet al., “Resolution-enhanced integral imaging microscopy that uses lens array shifting,” Opt. Express 17(21), 19253–19263 (2009).OPEXFF1094-4087http://dx.doi.org/10.1364/OE.17.019253Google Scholar

16. 

H. E. Ives, “Optical properties of a Lippmann lenticulated sheet,” J. Opt. Soc. Am. 21(3), 171–176 (1931).JOSAAH0030-3941http://dx.doi.org/10.1364/JOSA.21.000171Google Scholar

Biography

Junkyu Yim received his BS degree in 2012 and MS degree in February 2014, in information display from Kyung Hee University, Seoul, Republic of Korea. His current research interests focus on the three-dimensional display based on integral imaging technique.

Young Min Kim received his BS degree in 2011 and MS degree in February 2013, in information display from Kyung Hee University, Seoul, Republic of Korea. His current research interests focus on the three-dimensional display and visual fatigue of human vision related to three-dimensional display.

Sung-Wook Min received his BS and MS degrees in electrical engineering from Seoul National University, Republic of Korea, in 1995 and 1997, respectively. In August 2004, he received his PhD degree from his alma mater. Currently, he is a faculty member in the department of information display, Kyung Hee University, which he joined in 2007. He is interested in 3-D imaging and the advanced display system, especially based on the integral imaging technique.

Junkyu Yim, Young Min Kim, Sung-Wook Min, "Real object pickup method for real and virtual modes of integral imaging," Optical Engineering 53(7), 073109 (31 July 2014). http://dx.doi.org/10.1117/1.OE.53.7.073109
Submission: Received ; Accepted
JOURNAL ARTICLE
7 PAGES


SHARE
KEYWORDS
Image processing

3D image reconstruction

3D image processing

Integral imaging

Cameras

Imaging systems

3D displays

Back to Top