Real object-based integral imaging system using a depth camera and a polygon model

Abstract. An integral imaging system using a polygon model for a real object is proposed. After depth and color data of the real object are acquired by a depth camera, the grid of the polygon model is converted from the initially reconstructed point cloud model. The elemental image array is generated from the polygon model and directly reconstructed. The polygon model eliminates the failed picking area between the points of a point cloud model, so at least the quality of the reconstructed 3-D image is significantly improved. The theory is verified experimentally, and higher-quality images are obtained.


Introduction
Integral imaging is an autostereoscopic three-dimensional (3-D) display method that presents natural full-parallax and continuous-viewing 3-D images via common incoherent illumination.2][3] However, the quality of the final 3-D image is degraded due to several problems in the optical pickup process, such as distortion and contamination of the lens array, and imperfect matching of devices.5][6][7] However, most CGII methods generate the EIAs from virtual objects, not real ones.
Recently suggested is a simplified integral imaging pickup method that is a combination of both optical pickup and CGII techniques, which displays a 3-D visualization of a real-world object through a CGII algorithm. 8Here, a depth camera acquires the 3-D information of the real-world object (depth and color) at the same time, and a point cloud object space is reconstructed initially based on the acquired 3-D information.When the positions for all object points are set, the EIA is generated from the object space through a virtual lens array, i.e., a CGII algorithm, for each object point and is sent directly to the display device, while the lens array, which has the same specifications as the virtual lens array, reconstructs it as a 3-D image.By applying the image space parallel processing method using a graphics processing unit (GPU), [9][10][11][12][13][14] a real-time depth camera-based integral imaging display for real objects can be visualized. 15Although this system eliminates the optical issues of the lens array, the failed picking area (FPA) of the point cloud affects the final image quality.Note that an FPA is the empty area between the neighboring object points of the point cloud model, and these FPAs are visible in the generated EIA and the reconstructed 3-D image as black lines and/or portions.Also, there is another big issue in that the resolution of depth data is much lower than that of the color data, and a large amount of object information can be lost when the EIA is directly generated from the point cloud model.][18][19] Thus, in this paper, to improve the reconstructed real 3-D image quality and keep all the information of the object, a depth camera-based integral imaging system using a polygon mesh model, which is an FPA-free 3-D model with a solid and smooth surface, is proposed.Also, the proposed method accelerates the generation time of the polygon model by utilizing a polygonal-selection CGII technique due to the complexity of the polygon generation process.In the experiment, a higher-quality 3-D image based on the newly generated polygon model is obtained.As mentioned in the previous section, the earlier depth camera-based integral imaging system generates EIAs from a point cloud model.When the real depth and color data of the object are acquired through a depth camera, the point cloud model is reconstructed based on the distance-coded depth information and corresponding color information of the object.The desired light rays reflected from each object point pass through the center of each virtual elemental lens and are recorded as the pixels of elemental images, according to the general integral imaging pickup process, and the corresponding color information for each object point is matched to each pixel of an EIA. Figure 1 shows the schematic configuration of the EIA generation process from the point cloud model via the depth camera-based integral imaging system.However, during the EIA generation process for the point cloud model, some data for specific parts can be lost, and this happens often.This lost data are called the FPA, which is the empty space between the object points, and the FPAs significantly affect the reconstructed 3-D image quality.

Depth Camera-Based Integral Imaging System
Using a Polygon Model Figure 2 shows the overall scheme of the proposed system, which consists of three main processes: acquisition, polygon generation, and EIA generation/display.Generally, the depth camera acquires 3-D depth and color information of the real object at the same time by determining the object space for the entire depth area and the corresponding color data for the object space during acquisition.Based on the acquired data, a point cloud model that includes the real object space information is created.Then the coordinates of each pixel in the depth data and the corresponding pixel information for color data are transmitted to the next process, and the initial point cloud is converted into the polygon model by filling the empty spaces between object points, i.e., the FPAs.In the final process, the EIA is generated for the newly generated polygon model, and the 3-D image is reconstructed from the EIA.Generally, the color information of the real object, i.e., the resolution of color sensor of the depth camera is much larger than the depth sensor.Clearly, if the EIA is generated for the initially acquired data, much of the information about the object can be lost, where the real 3-D information of the object is stored as the specific coordinates, as shown in Fig. 3.
The polygon model has a solid and smooth outer surface compared with a point cloud model, so it can be an effective solution for reconstructed image quality of the depth camerabased integral imaging system.Unlike the point cloud model, the polygon model consists of a many vertices and triangular mesh elements instead of points.First, each pixel of the depth information in Fig. 4(a) is matched up to a corresponding pixel in the color information, as shown in Fig. 4(b), where the dark circles represent the corresponded pixels (visible in both color and depth information), and the white circles represent the noncorresponded pixels (visible only in color information).Basically, if a conventional Delaunay triangulation algorithm has been applied, it would have generated the polygon model for the points with corresponding color information after the corresponded pixels are detected, as shown in Fig. 4(b). 20When the number of object points is given by n, the entire process is Fig. 1 The EIA generation process of the depth camera-based integral imaging system using a point cloud model.performed in Oðn 2 log nÞ computational complexity.The Delaunay triangulation process requires a long processing time for a larger n.So, in this paper, a simple triangulation method is proposed, and it is performed by arranging the vertices of the polygon model in grid form directly from the depth information.The triangulation results for depth information can be preserved as they are in color information.For example, from Figs. 4(a) and 4(b), when assuming that three neighboring points of depth information (i, , and (l þ 4, k þ 3) in color information.The entire surface of the polygon model is generated using the neighboring vertical and/or horizontal vertices of the depth information, and it can save a great deal of processing time due to having a lower complexity than color information.
Assume that the nearest pixels in the depth information are determined like V 1 , V 2 , V 3 , and V 4 in Fig. 5. Clearly, depth information is used to generate a 3-D polygon model, and color information is used as texture mapping data on the generated 3-D polygon model.Two polygons consisting of vertices (V 1 , V 2 , V 4 ) and (V 2 , V 3 , V 4 ) and their corresponding texture coordinates are obtained; then the information is added into an array of polygons and texture coordinates.A set of polygons is generated from all the nearest pixels in an inputted depth image as shown in Fig. 5(a).Figure 5(b) shows the 3-D polygon modelgeneration process from the depth and color information acquired from the depth camera.
On generating a 3-D polygonal model, we consider the coordinates of each point of a polygon.Note that the polygon model is made up of vertices that are defined by the coordinates of the image space.We assume that DI w and DI h are the width and height of the color information, i.e., a color image, and DI d is the depth information acquired by the depth sensor.Obviously, the (x, y, z) coordinates for all vertices are set to the initial values of DI w , DI h , and DI d ranges, respectively.However, these coordinates need to be converted into a new coordinate system whose origin is located at the center of object space, as shown in Fig. 6, to match to the virtual lens array and normalize the polygon model to the depth range, which a lens array can properly acquire.
The specific vertex (D x , D y , D z ) defined in the image coordinate system can be transformed to the new coordinate system Oðx; y; zÞ on the virtual lens array space as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 3 2 6 ; 3 1 8 where OP is the distance between the virtual lens array and corresponding orthogonal plane that can occur in each depth value of object points, f LA is the focal length of the virtual lens array, g is the gap between the EIA plane and the virtual lens array, z CDP is the central depth plane that is the distance between the central plane of the polygon model and the virtual lens array, and P D is a pixel pitch of the EIA plane or display device.The EIA is generated from this depthmatched polygon model, the details of which are illustrated later.
Figures 7(a) and 7(b) show the rendering results, with a point cloud representing a scene captured from a depth camera, and with the 3-D polygon model generated by the proposed algorithm, respectively.Figure 7(a) shows an example  of the point cloud model after color information is plated on the depth information, an enlarged detailed representation of a region marked as a yellow rectangle, and a rotated representation of the point cloud model to present it more precisely.In Fig. 7(b), the texture mapping result applied to the generated polygon model is presented.
In the EIA generation stage, the EIA is generated for the newly generated depth-matched polygon model, and a fast computation method is applied that generates and displays the entire EIA within the shortest possible time.Here, a visible image from each elemental lens with respect to the polygon model is generated via a single thread using the virtual ray-tracing method; thus, the entire EIA generation time can be reduced since all threads can work simultaneously for every elemental lens.The entire detailed process of EIA generation from the polygon model is shown in Fig. 8.
The EIA generation process consists of three substages: preprocessing, EIA generation, and display.In the preprocessing, to prepare the pickup process for elemental images, a virtual space that contains the polygon model, a virtual lens array, and an EIA plane is built, where specifications of the virtual lens array are set by the user.The position of each elemental lens inside the virtual space is calculated as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 3 2 6 ; 4 6 9

EL
x ¼ From Eq. ( 2), ELðx; yÞ is the position of each elemental lens, LA w and LA h are the width and height of the entire virtual lens array, and LP w and LP h are the width and height of any (i, j)'th elemental lens.Equation ( 2) is carried out only once in the preprocessing substage of EIA generation.The next substage is the EIA pickup that generates the elemental images based on the preprocessing stage from the polygon model.It creates the same number of threads as the number of elemental lenses, and a ray group is generated that has the same number of rays as the number of pixels of each elemental lens within each thread, as shown in Fig. 9.Each ray passes through the center of an elemental lens from the EIA plane and visualizes only the plane information intersecting with the corresponding ray, where this process is run simultaneously for every ray in every thread; at the least, the huge computation processing time is greatly reduced.
Finally, in the display substage, the generated EIA is projected on the display device, while the lens array reconstructs it as a 3-D image for the observer.An example of an EIA generated from the polygon model is shown in Fig. 10.Here, it can be seen that the FPAs are invisible in the EIA generated from the polygon model, as shown in Fig. 10(b), whereas too many FPAs are visible in the point cloud-based EIA, as shown in Fig. 10(a).

Experimental Results
The specifications of the experimental object and devices are listed in Table 1, and the experimental environment is presented in Fig. 11.
The depth camera acquires depth information with 512×424pixels, and color information with 1920× 1080 pixels, and the polygon model consisting of 217,088 vertices (0.2 MB) is generated based on the acquired data.We prepared three kinds of color images, from simple to complicated, to provide the experimental results of the proposed method, as shown in Fig. 12(a).The EIAs generated from the initial point cloud and the newly generated polygon models are presented in Figs.12(b) and 12(c), respectively.The proposed method generates a 28 × 28 elemental image for each lens and generates the entire EIA resolution at 840 × 840 pixels by considering the lens array size, where the entire size of lens array is 150 × 150 mm 2 .Compared with the EIAs generated directly from point cloud models, the EIAs for polygon models are FPA-free, which means that the EIA does not have the empty areas (i.e., black lines in the image), and the 3-D information of the real object can be exactly recorded to the EIA.Therefore, it can be verified that the polygon model has an FPAfree solid outer surface, whereas the point cloud model has too many FPAs.Fig. 9 The EIA generation process using the ray group.
Figure 13 shows the numerically reconstructed 3-D images using the computational integral imaging reconstruction algorithm 21  There are many FPAs that appear as black lines in the images in the odd column, but on the other side, there are FPA-free images in the even column.From these images, it can be verified that a quality difference exists between the images according to the depth of field.Especially for the shoulder, silhouette, and face areas, the image quality degrades when the depth of field increases.
To measure the 3-D image quality and compare the previous point cloud-based system to the proposed polygonbased one, peak signal-to-noise ratio (PSNR) is utilized for the three data cases (test 1 for 32 mm, test 2 for 43 mm, and test 3 for 53 mm), as shown in Fig. 12     17.9, 18.3, and 19.6 dB for a 53-mm depth plane.Thus, we can see that the proposed method successfully improves the reconstructed image quality of the depth camera-based integral imaging system at all the allowable depth planes when compared with the conventional case.
4 Conclusions A polygon model-based quality-enhanced integral imaging display system is proposed and implemented.The initial pointcloud model is generated based on the acquired depth and color information of real-world objects through a depth camera, and the polygon model is converted from the point cloud by applying a proposed triangulation algorithm for each object point.The final reconstructed image has better quality than a point cloud model-based method, in that the PSNR value is higher by ∼0.5 to 2.0 dB.However, the proposed method cannot satisfy a real-time display due to the huge computation time for conversion from real 3-D data to the virtual 3-D object.
To provide high-speed computation, intermediate view image generation and/or GPU-based parallel processing algorithms are required because these kinds of methods are able to shorten the time needed to generate the entire image.

Fig. 2
Fig.2An overall procedure of the proposed method.

Fig. 3
Fig.3The resolution difference between (a) the color and (b) the depth information of a real object acquired by the depth camera.

Fig. 4
Fig. 4 The principle of the grid-based 3-D polygon model generation: (a) the proposed triangulation method from neighboring points of depth information to vertices and (b) application of the simple triangulation process for color information.

Fig. 5
Fig. 5 (a) Triangulation from nearest points of depth information and (b) generation step for a 3-D polygon model with depth and color data.

Fig. 6
Fig.6Geometry of transformed coordinate system on the virtual lens array space.

Fig. 7
Fig. 7 Rendering results difference between (a) a point cloud and (b) a polygon model generated by the proposed algorithm.

Fig. 8
Fig.8EIA generation process for a newly generated polygon model.
Figure13shows the numerically reconstructed 3-D images using the computational integral imaging reconstruction algorithm21 for the generated point cloud-based and polygon-based EIAs.To present the real 3-D information of the object, the images captured from different viewpoints along the different reconstruction distances are presented.In the , each of which consists of the color and depth information, the corresponding point cloud, and the polygon-based EIAs.PSNR for test 1, test 2, and test 3 are represented with blue, orange, and gray colors, respectively.The measured PSNR values for the polygonbased model and the point cloud-based model are presented in Figs.14(a) and 14(b), respectively.Here, PSNR values are measured for different depth planes (32, 43, and 53 mm) in each reconstruction.PSNR values from three polygonbased models in Fig. 14(b) were measured at ∼21.2, 22.1, and 23 dB for a 32-mm depth plane at 21.9, 22.2, and 23 dB for a 43-mm depth plane and at 21.8, 22.6, and 22.9 dB for a 53-mm depth plane; the PSNR values for point cloud-based cases in Fig. 14(a) were measured at ∼18.3, 18.7, and 19.2 dB for a 32-mm depth plane, at 18.1, 18.4, and 19.8 dB for a 43-mm depth plane, and at

Fig. 10
Fig. 10 The comparison between the EIAs from point cloud and polygon models: (a) EIAs generated from the point cloud includes many FPAs and (b) for the polygon model, the EIA is generated as FPAfree.

Fig. 12 (
Fig. 12 (a) For the real object from simple to complex, (b) the point cloud-based EIA is generated, where the FPAs are visible as black lines, and (c) the clearer EIA generated (without FPA) from the polygon model.

Fig. 13
Fig. 13 The reconstructed 3-D image generated on the different depth planes for given three objects: (a) basic single object, (b) complicated single object, and (c) multiple objects.

Table 1
Specifications of experimental devices and EIA Fig.11The system implementation for the proposed method.