1.

Introduction

Among research on diverse three-dimensional (3-D) displays, there have been a lot of studies on the autostereoscopic 3-D display^1–10 which provides multiview 3-D images. The basic principle of this multiview 3-D display is to provide more than two views by making each view spatially distributed within the viewing zone using an optical structure. Then, two different views among many views can enter into each pupil of an observer, so that the one can recognize 3-D images seen differently according to the observing position. Not only that but also a motion parallax can be provided from many different views along the horizontal direction of viewing zone. In order to provide different view images according to a position in the viewing zone, any type of optical plate such as a parallax barrier and a lenticular lens sheet is conventionally used by being attached onto a flat two-dimensional (2-D) display panel. Those optical plates play a role of controlling the light path so that many different views can be properly separated in the viewing zone at optimum viewing distance (OVD). That is how to have an observer able to recognize a 3-D scene from different images coming into each eye. For this, different color information corresponding to each view image should be assigned to subpixels on the flat panel display, for example, a 2-D LCD display. In general, such optical plates can be slanted to be attached onto the 2-D display panel. In this case, a group of subpixels on a line with a slanted angle is supposed to correspond to one of the multiview images.

Therefore, in order to make the images for each view spatially separated at OVD, an optical plate must be accurately attached to the 2-D display panel without any misalignment. Otherwise, 3-D image quality can be significantly degraded with some increased crosstalk. Practically, however, it is very hard to align the optical plate accurately to the 2-D display panel. In the process of assembly, the occurrence of some misalignment is very likely. To reduce 3-D image distortion in a misaligned 3-D display, there have been some studies^11–14 that suggested some methods using an image processing to resolve the image quality problem from the mismatch of an optical plate.

In this paper, however, we propose an alternative correction method for distorted 3-D images caused by the misalignment of an optical plate without changing the pixel mapping of view number assigned to each subpixel for a designed angle. It will be shown that the correction can be done by simply skewing 3-D objects with appropriate angles according to the amount of misalignment. In the first section, it is introduced how the misalignment can result in the distortion of 3-D image. By analyzing the distortion phenomenon, the formula that determines a skewing angle to correct the distorted 3-D image is derived in the second section. In the next section, it will be shown with some examples that distorted 3-D images in a misaligned 3-D display are perfectly corrected with skewing angles given by the formula. In conclusion, we summarize our result with some comments.

2.

Distortion Phenomenon of 3-D Images Caused by Angular Misalignment

Misalignment can be categorized into three cases which are the angular, axial, and lateral misalignments.¹⁵ The angular misalignment implies the mismatch between the designed and assembled angles of a slanted optical plate. The axial misalignment can occur when the assembled gap between the optical plate and 2-D display panel is different from the designed gap. The lateral misalignment comes from the vertical or horizontal translation of the optical plate. In general, the lateral misalignment results in shifting the viewing zone to the left or right from the center of the display. But it is relatively easy to correct that kind of misalignment. We can have a centered viewing zone by simply shifting a pixel mapping entirely with some proper amount. And the axial misalignment can result in a change of the OVD position, particularly for the barrier type of 3-D display. For the lenticular type of 3-D display, due to the mismatch between the focal length of lenticular lens and the gap the crosstalk can be increased, so that the 3-D image quality can be aggravated. However, this can be also controlled by using the gap glass with proper thickness. Compared to those misalignments, the angular misalignment has more something to do with 3-D image quality since it can happen very often in a manufacturing process and it is also difficult to resolve the 3-D image quality problem. It can not only deteriorate 3-D image quality from the increase of crosstalk but can also make the 3-D image remarkably distorted. Even though there could be some image processing^11–14 to be able to make the 3-D images better by reducing some crosstalk, it is rather hard to completely fix the image quality problem since the mismatch of a mapping rule to subpixels has different amounts on the entire 2-D display. In this paper, we will only focus on the distortion phenomenon of 3-D images caused by the angular misalignment and suggest a method to correct the distorted 3-D images without changing a mapping rule on subpixels of a 2-D display. For this, let us first show an example about how 3-D images can be distorted by the (angular) misalignment. If there is no misalignment between an optical plate and a 2-D display panel, the 3-D image must be correctly expressed without any distortion as in Fig. 1(a). However, when there is some misalignment, the distortion phenomenon of 3-D image happens as in Fig. 1(b). As seen in Fig. 1, the distortion of the 3-D image by the misalignment appears as a tilted image. The reason why an object with some depth in 3-D images is tilted is because the misalignment makes 3-D images of different views seen at the same time even though the observer is at the OVD position of one view. Therefore, the objects become more tilted when not only the amount of misalignment but also the expressed depth increases. That is, the object with more disparity is more tilted. It can be also checked in Fig. 1 that the objects with different depths are tilted with different amounts. For the case of a well-aligned 3-D display, but not perfectly, as shown in Fig. 1(a), the 3-D image shows that the vertical sticks are all upright. In the case of a misaligned 3-D display, however, it is shown in Fig. 1(b) that all vertical sticks are not upright but tilted except for the stick with zero depth. Note that the tilted direction of objects is completely opposite depending on the sign of an expressed depth. In our convention, a negative depth means an object with a negative disparity, which is seen as located in front of the 2-D display panel.

Fig. 1

Example of a distorted phenomenon of 3-D image: the first stick from the left has $-200\mathrm{mm}$ depth, the second one from the left has $-100\mathrm{mm}$ depth, and the stick at the center has no depth, while the first stick from the right has $+200\mathrm{mm}$ depth and the second one from the right has $+100\mathrm{mm}$ depth. (a) Right expression of 3-D image in a well-aligned 3-D display and (b) distorted 3-D image in a misaligned 3-D display.

3.

How to Correct Distorted 3-D Images

In order to correct the distortion of tilted 3-D images, it can be simply considered to skew 3-D objects in the opposite direction to the tilted one. For this, it is necessary to know how much the objects should be skewed and which parameters have something to do with the skewed amount of them. In this section, we will derive an equation which determines the amount of skewing an object, depending on the amount of misalignment and the depth of the object. Let us think more of the reason why 3-D images are tilted in detail. As mentioned before, because of the misalignment we are supposed to see not only the subpixels for a certain view but also the subpixels for different views at the same time even at OVD. Suppose that total view number is $N_{\mathrm{tot}}$ and the optical plate is slanted with the angle $\alpha$ as in Fig. 2. When the designed angle is $\alpha$, it can be generically expressed in terms of the vertical and horizontal number of subpixels as follows:

Fig. 2

Scheme for some parameters related to a designed angle and an assembled angle.

Fig. 3

Scheme for the definition of total misalignment amount $\mathrm{\Delta }M$ in a misaligned 3-D display.

Now, in order to find the skewed amount to correct the distortion of tilted 3-D images, let us consider a vertical line with a negative depth of which the two end points $A$ and $B^{\prime }$ corresponding to the same $p$’th view has the vertical distance $H$ in a 2-D display panel, as in Fig. 4(a). For simplicity, it is supposed that the natural number $k$ is satisfied with the condition, $k=k^{\prime }{N}_h+1$ with a natural number $k^{\prime }$. If the alignment is perfect, two subpixels $A$ and $B^{\prime }$ for the $p$’th view will be seen together at the OVD position of its view. But, when there are some misalignment $\mathrm{\Delta }M$, not $B^{\prime }$ but $C^{\prime }$ corresponding to $(p-m\mathrm{\Delta }M)$’th view is seen together with $A$ at the same time, so that two subpixels being seen are not on the vertical line but on a tilted line. When the displacement of $C^{\prime }$ from $B^{\prime }$ is given by $x$ in a 2-D display panel, the resulting 3-D image is tilted as the red line in Fig. 4(a). For the case of a negative depth, the geometric relation between camera interval $l$ in 3-D space and the displacement $x$ on the 2-D display is described in Fig. 4(b) where the $X$ plane just represents the disparity zero plane of the 2-D display panel and the $X^{\prime }$ object plane indicates a sliced 2-D plane with a negative depth in the 3-D space. To obtain the corrected 3-D images, the $B^{\prime }$ point should be skewed in the opposite direction with the same amount of the displacement $x$ before rendering each 3-D image for all cameras. In other words, in Fig. 4(a) the $B^{\prime }$ point should be shifted to the left with $x$ amount in order to make it seen with A on a vertical line. That implies the line object on the $x^{\prime }$ object plane should be skewed with the amount of $x^{\prime }$. However, since the displacement has different values according to the vertical position on the $X$ plane, it depends on the height of an object on the $X^{\prime }$ plane. Therefore, it is necessary to introduce an invariant quantity regardless of an object’s height. For this, the skewing angle ${\theta }_{-}$ is introduced as represented in Fig. 5(a). Then, using the geometric relations in Fig. 4, the skewing angle can be obtained as

Fig. 4

Geometric scheme for the correction of the distorted 3-D image with a negative depth: (a) tilted phenomenon for an upright straight line and (b) geometric relation among the camera interval at the OVD position, $X$ plane on the 2-D display panel, and $X^{\prime }$ object plane with a negative depth in 3-D space.

Fig. 5

Geometric relation among the $X$ plane, $X^{\prime }$ object plane, and OVD position in 3-D space: (a) $X$ plane with no depth, (b) sliced object plane $X^{\prime }$ with a negative depth in 3-D space, and (c) side view of the $X$ and $X^{\prime }$ planes in 3-D space.

Likewise, we can also obtain a similar equation for the case of a 3-D object with a positive depth by using trigonometry. Therefore, regardless of the sign of depth, the skewing angle $\theta$ for the correction of a distorted 3-D image can be finally obtained as follows:

Fig. 6

Skewed direction of a sliced object plane $X^{\prime }$ in 3-D space according to the sign of $\theta$ value: (a) $\theta >0$ case and (b) $\theta <0$ case.

Now, let us comment on how to measure the amount of a misalignment $\mathrm{\Delta }M$. Even though there must be various ways to find out how much an optical plate is misaligned, some examples about how to measure the amount of misalignment $\mathrm{\Delta }M$ are left in Appendix A. In summary, one method is to use the subpixels of a 2-D display as a coordinate. Basically, if two subpixels are seen together at the same time at a fixed position of OVD, the displacement between them in vertical and horizontal directions can be found in terms of the number of subpixels. Another way is to make use of the fact that the resulting 3-D images are tilted if there is some misalignment. So, the test patterns with some skewed images corrected for various $\mathrm{\Delta }M$ can be used. If a skewed image among those test patterns is upright as a corrected one, the corresponding misalignment $\mathrm{\Delta }M$ is nothing but the amount of the misalignment of the optical plate in a 3-D display. Alternatively, some test patterns made for various attached angles can be also used. Similarly, by finding the resulting 3-D image is not tilted but upright the actual attached angle of an optical plate can be easily found. Then the amount of the misalignment $\mathrm{\Delta }M$ can be obtained from the difference of the designed and assembled angles. Note that even if $\mathrm{\Delta }M$ is measured as not an integer but a rational number, Eq. (5) still holds.

4.

Examples for Corrected 3-D Images through Experiments

In this section, we will show some experimental results that distorted 3-D images can be corrected with Eq. (5). To show some examples, a misaligned 3-D display of which the amount of the misalignment is $\mathrm{\Delta }M=+14$ is used. The designed angle of the multiview 3-D display is $\mathrm{ArcTan}(1/9)$, which implies $m=3$ and $n=1$ in the form of Eq. (1). And the vertical pitch of the subpixel is $b=0.158\mathrm{mm}$ and the vertical resolution of the 2-D display is 2160.

In Fig. 7 some vertical sticks with different depth are used as 3-D objects. Figure 7(a) represents distorted 3-D images because of the misalignment, so that the vertical sticks are seen tilted with different amounts according to the depth. By skewing the objects with different skewing angles, however, it can be shown in Fig. 7(b) that the tilted vertical sticks are all corrected and upright completely. The skewing angles for each object are obtained from Eq. (5).

Fig. 7

Example of correction of 3-D image in a misaligned 3-D display: the first object (stick) from the left is at $-200\mathrm{mm}$ depth, the second one is at $-100\mathrm{mm}$ depth, and the third one has no depth. The fourth one is at $+100\mathrm{mm}$ depth and the fifth one has $+200\mathrm{mm}$ depth. (a) Before correction and (b) after correction by skewing 3-D objects.

Another corrected example is shown in Fig. 8. Before correction, Fig. 8(a) represents the distortion of 3-D images in which it can be also checked that each card is tilted with different amounts on account of different depths. However, after the correction by skewing all 3-D objects with the determined angles from Eq. (5), it is shown in Fig. 8(b) that all the distorted 3-D images are completely corrected.

Fig. 8

Example of correction of 3-D image in a misaligned 3-D display: The 11 cards have all different depths from $-200$ to $+200\mathrm{mm}$ with 40 mm depth gap in order. So, card number 1 has $-200\mathrm{mm}$ depth, card 6 has no depth, and card 11 has $+200\mathrm{mm}$ depth. The background picture is located at $+250\mathrm{mm}$ depth. (a) Before correction and (b) after correction by skewing 3-D objects.

The other example for the correction of some other figures is shown in Fig. 9. Even though circles are used as 3-D objects, it appears as ellipses because of the misalignment. In Fig. 9, it is shown that the distorted 3-D images as ellipses are all corrected as exact circles.

Fig. 9

Example of correction of 3-D image in a misaligned 3-D display: the left one has $-220\mathrm{mm}$ depth, the center one has no depth, and the right one has $+220\mathrm{mm}$ depth. (a) Before correction and (b) after correction by skewing 3-D objects.

5.

Conclusion

In this paper, an alternative method to correct distorted 3-D images has been suggested. Since the resulting tilted 3-D images as distortion are attributed to the misalignment of the optical plate on a 2-D LCD display panel, they can be corrected by skewing 3-D objects in the opposite direction to tilted ones, without changing a mapping rule. To determine the amount of skewing angles for the correction of distorted 3-D images, we have derived the formula depending on the amount of misalignment and the depth of objects. By using the formula we have showed that the distorted 3-D images can be corrected by skewing the 3-D objects with different angles according to their depths when they are rendered by multiple cameras. With some examples, it has been shown that the skewed objects can be expressed correctly in a misaligned multiview 3-D display. In other words, if the 3-D space is skewed with some proper amount according to the depth, the distortion of 3-D image can be completely corrected. Practically, since skewing 3-D space is nothing but a coordinate transformation of the 3-D space, distorted 3-D image in a misaligned 3-D display can be also corrected by manipulating 3-D image data obtained from a depth camera. This method to skew 3-D space according to depth can resolve the problem of distorted 3-D images by angular misalignment regardless of no matter how much the misalignment occurs, which implies that it is not needed to align perfectly an optical structure in a multiview 3-D display system. However, since angular misalignment can have an influence on the formation of viewing zone, too much misalignment can lead to the reduction of the viewing zone.

In order to check how well distorted 3-D images are corrected, the tilted image with a certain depth can be compared to the image with zero depth since the zero-depth image is not distorted by the misalignment. If they are observed parallel, it can be considered that the distorted image is corrected completely up to the perceptual limit. Since the image with more depth is tilted more, the image with enough depth can facilitate checking the level of correction unless the 3-D image quality degraded by some crosstalk due to depth expression impedes the observation. Since the tilted images can be perfectly corrected and upright with the exact skewing angles theoretically given through the accurate measurement of misalignment, unparalleled results can imply the inaccurate measurement of the misalignment. There could be various ways to measure quantitatively how parallel they are. As a simple example, by drawing lines along the images from the taken pictures what can be checked is how much they are unparalleled. To decide the parallelism more objectively, however, it might be needed to find another method to assess quantitatively how well distorted 3-D images are corrected. Since the correction of tilted 3-D images in a misaligned multiview 3-D display has been performed in this paper by skewing 3-D objects in 3-D space, the environment where each image for multiple cameras is rendered is not same so that the corrected output image is completely different to the original one. So, for the assessment on correction it could be inadequate to compare those two images with a typical image quality assessment metric such as structural similarity or peak signal-to-noise ratio. As mentioned, since it would rather be related to the accurate measurement of misalignment, the assessment on the accuracy of a measured misalignment would be able to reflect the level of correction more accurately in a quantitative way.