Integral imaging without image distortion using micro-lens arrays with different specifications

Abstract. We propose an integral imaging in which the micro-lens array (MLA) in the pickup process called MLA 1 and the micro-lens array in the display process called MLA 2 have different specifications. The elemental image array called EIA 1 is captured through MLA 1 in the pickup process. We deduce a pixel mapping algorithm including virtual display and virtual pickup processes to generate the elemental image array called EIA 2 which is picked up by MLA 2. The three-dimensional images reconstructed by EIA 2 and MLA 2 do not suffer any image scaling and distortions. The experimental results demonstrate the correctness of our theoretical analysis.


Integral imaging without image distortion using micro-lens arrays with different specifications 1 Introduction
Integral imaging (II), which was originally called integral photography, is one of the most attractive techniques in the field of three-dimensional (3-D) displays.It has many advantages over other 3-D display techniques.For example, it provides continuous viewpoints within a specific viewing angle in both horizontal and vertical directions and does not need any special glasses for viewers. 1 Compact system configuration is another advantage which makes it a practical alternative to volumetric or holographic 3-D displays.So far, researchers have overcome the problems of low 3-D resolution, 2,3 limited depth of field, etc. [4][5][6][7][8][9][10][11] How to make the II 3-D display accepted by home users is an urgent issue.In the conventional II, the parameters of micro-lens arrays (MLAs) used in both pickup and display processes should be identical.In practical use, we cannot expect all II 3-D display systems to have MLAs with the same specifications, and we could not prepare various sets of elemental image arrays (EIAs) for different II 3-D display systems.To be an ideal 3-D television technique, we have to make the EIA compatible with different II 3-D display systems for a huge number of home users.
The theoretical analysis on scaling rule between the primary and secondary pictures taking was firstly carried out by Okoshi. 12His analysis was based on the assumption that the elemental images (EIs) are not scaled when shifted with respect to the optical axes of the corresponding micro-lenses.Park et al. discussed the scaling effects of the optical parameters difference between the pickup and display processes. 13hey showed that the lateral magnification is given by the micro-lens size ratio and the longitudinal magnification is given by the focal length ratio.Then, they proposed a ratio-conserving scaling to eliminate distortion of the 3-D image in which the EIAs are scaled using the focal length ratio. 14But these methods cannot cope with a change in the number of micro-lenses.The EIAs obtained in the pickup process are digitally analyzed and full 3-D information of the object are extracted, and then the extracted 3-D information is transmitted to display systems, whose numbers and parameters of micro-lenses are entirely different for the pickup process. 15However, stereo matching is needed to detect the disparity between two or more EIs and the quality of the generated 3-D image will be degraded.The scaling was also achieved by controlling the spatial ray sampling rate of EIA in the pickup process 16 and a digital magnification method using interpolation theory was proposed to increase the spatial ray sampling. 17The intermediate-view reconstruction technique using the same interpolation theory was used to increase the number of EIs (Ref.18).But the parameters of the micro-lens are not changed in these techniques.Smart pixel mapping algorithm is proposed by Martínez-Corral et al. 19 to resolve the pseudoscopic problem of the reconstructed 3-D image, which is also resolved by Jung et al. recently. 20n this paper, we propose an II without image distortion using MLAs with different specifications in the pickup and display processes.The deduced pixel mapping algorithm not only resolves the pseudoscopic problem, but also generates the EIA with different lens specifications, and the reconstructed 3-D images do not have any distortions.
2 Principle of the Proposed II Figure 1 shows the schematic of the proposed II which has different specifications of MLAs in the pickup and display processes.In the pickup process, as shown in Fig. 1(a), the MLA and the EIA are called MLA 1 and EIA 1, respectively.The pitch and the focal length of MLA 1 are p 1 and f 1 , respectively, and MLA 1 contains M 1 × N 1 micro-lenses.EIA 1 is recorded on the rear focal plane of MLA 1 and the pitch of EIA 1 is the same with that of MLA 1.In the display process, as shown in Fig. 1 Figure 2 shows the schematic of the proposed pixel mapping algorithm which includes the virtual display process and the virtual pickup process.Since the pickup process and the virtual display process form the standard configuration of II, EIA 1 and MLA 1 reconstruct a 3-D image that maintains the original size and location of the 3-D object, but with the reversed depth.But the pseudoscopic problem can be resolved by the pixel mapping algorithm because the combination of the virtual display process and the virtual pickup process is a modified version of the two-step pickup process.In the virtual pickup process, MLA 2, which is the same as the one used in the display process in Fig. 1(b) and which has different specifications from MLA 1, picks up the depth-reversed 3-D image.The distance between MLA 1 and MLA 2 is L. EIA 2 that has different specifications from EIA 1 can be obtained on the rear focal plane of MLA 2, and EIA 2 and MLA 2 have the same pitch of p 2 .The resolutions of EIs in EIA 1 and EIA 2 are both r × r.
In practicae, the virtual display and virtual pickup processes are carried out by mapping all the pixels in EIA 1 to EIA 2 through the following mathematical relationships.As shown in Fig. 2, in the m'th row and the n'th line EI of EIA 1, a pixel in the i'th row and the j'th line is denoted as I 1 ðm; nÞ i;j .The rays emitted from the pixel I 1 ðm; nÞ i;j are refracted by the micro-lenses in MLA 1 and MLA 2, and then arrive at the i'th row and the j'th line pixel of the m'th row and the n'th line EI in EIA 2. The pixel in EIA 2 is denoted as I 2 ðm 0; n 0Þ i 0;j 0 .EIA 2 can be generated by using the following mathematical relationships: (1) where the function round ð•Þ rounds a number to the nearest integer.When i or j is bigger than r, the pixel should be abandoned to eliminate the overlapping between adjacent EIs.In this way, in loop m from 1 to M 1 , n from 1 to N 1 , i from 1 to r, and j from 1 to r, all the pixels in EIA 1 are mapped to the rear focal plane of MLA 2, and an EIA 2 that has different specifications from EIA 1 is generated.
The distance L between MLA 1 and MLA 2 determines the depth of the reconstructed 3-D image.Assuming that in the pickup process, the distance between the 3-D object and MLA 1 is l a , the depth of the reconstructed 3-D image in the display process is When l a ¼ L, the 3-D image will be displayed on the MLA 2 plane, and when l a > L or l a < L, the 3-D image will be displayed behind or in front of the MLA 2 plane.
When the light rays emitted from the pixel in the adjacent EI in EIA 1 are refracted by MLA 1 and MLA 2 and arrive at EIA 2, the crosstalk pixel is produced, as shown in Fig. 3.So the pitches and the focal lengths of MLA 1 and MLA 2 should satisfy Eq. ( 7) to avoid the crosstalk pixel The uninformed pixel will be caused when pixels in EIA 2 have no corresponding pixels in EIA 1, as shown in Fig. 4.So the numbers of EIs in EIA 1 and EIA 2, M 1 × N 1 and M 2 × N 2 , should satisfy Eqs. ( 8) and ( 9) to avoid the uninformed pixels where the function ceilð•Þ rounds a number to the largest integer.

Experiments and Results
In this experiment, a 3-D scene which consists of three plane images with different depths is built up as shown in Fig.

Conclusions
In this paper, we propose an II in which MLA 1 in the pickup process and MLA 2 in the display process have different specifications.The pixel mapping algorithm that functions as a converter to transmit the pixels from EIA 1 to EIA 2 not only resolves the pseudoscopic problem but also generates the EIA with different lens specifications.We also deduce the mathematical relationship between EIAs 1 and 2, and the relationships between the parameters of EIAs 1 and 2, and between MLAs 1 and 2. As long as the parameters of MLA 1 and MLA 2 satisfy the relationships in Eqs. ( 7)-( 9), the parameters of MLA 2 and EIA 2 can be selected arbitrarily, hence different EIAs 2 can be generated from EIA 1 for different II 3-D display systems.The experimental results demonstrate that the reconstructed 3-D images in the proposed II maintain the original lateral and longitudinal sizes of the 3-D object without any scaling and distortion.The proposed II could be an ideal candidate for 3-D television broadcasting in the future.
(b), the MLA and the EIA are called MLA 2 and EIA 2, respectively.The pitch and the focal length of MLA 2 are p 2 and f 2 , respectively, and MLA 2 contains M 2 × N 2 micro-lenses.EIA 2 generated by the pixel mapping algorithm has different specifications from EIA 1, and the 3-D image reconstructed by EIA 2 and MLA 2 maintains the original size and location of the 3-D object without any distortions.

Fig. 1
Fig. 1 Schematic of the proposed II using MLAs with different specifications in (a) pickup and (b) display processes.

Fig. 2
Fig. 2 Schematic of the proposed pixel mapping algorithm.(a) Virtual display process and (b) virtual pickup process.Fig.3 Crosstalk pixel caused by adjacent EI in EIA 1.

Fig. 3
Fig. 2 Schematic of the proposed pixel mapping algorithm.(a) Virtual display process and (b) virtual pickup process.Fig.3 Crosstalk pixel caused by adjacent EI in EIA 1.
5. A camera array is used to simulate MLA 1.The Z-axis shows the distances between the characters and the camera arrays, and they are 60, 100, and 150 mm, respectively.Three experiments using the proposed II, conventional II without scaling, and straightforward scaling II in Ref. 11 are carried out.The specifications of MLA 1 and MLA 2 are listed in Table 1 in which the parameters of MLA 1 and MLA 2 in the proposed II satisfy the relationships in Eqs.(7)-(9).Three EIAs 2 obtained using the three II methods are shown in Figs.6(a), 6(b), and 6(c), respectively.The depth-based computational II reconstruction is implemented to reveal explicitly the cross sections of the reconstructed 3-D images along the longitudinal direction, so that the longitudinal magnification of the 3-D image can be readily determined.Since the distance L between MLA 1 and MLA 2 is 100 mm, the reconstructed undistorted 3-D images should be located at the depths of −50, 0, and 40 mm, respectively.As shown in Figs.7(a) and 7(b), the 3-D images reconstructed by our proposed II have the same depths without the pseudoscopic problem as the ones reconstructed by the conventional II and their longitudinal magnifications are both 1.However, using the straightforward scaling II as shown in Fig. 7(c), the reconstructed 3-D images have the longitudinal magnification of f 2 ∕f 1 ¼ 0.86.

Fig. 4
Fig. 4 Comparisons between the numbers of EIs in EIAs 1 and 2.

Fig. 6
Fig. 6 EIAs 2 generated by the (a) proposed II, (b) conventional II without scaling, and (c) straightforward scaling II.

Fig. 7
Fig. 7 3-D images at different depths reconstructed by the (a) proposed II, (b) conventional II without scaling, and (c) straightforward scaling II.

Fig. 8
Fig. 8 3-D images reconstructed by the (a) proposed II, (b) conventional II with no scaling, and (c) straightforward scaling II.

Table 1
Specifications of MLAs 1 and 2 in the experiments.