2.1.

Acquisition Models for Monoscopic Panoramas

There are two main models for the acquisition of monoscopic panoramas: a singular viewpoint (SVP) model and a nonsingular viewpoint (non-SVP) model, also known as polycentric panoramic model. Any camera or acquisition technique available to produce monoscopic omnidirectional imagery can be classified into one of these two models.

In the SVP acquisition model, for any gazing direction in azimuth (camera’s panning direction), there is a unique projection center that marks a single convergence point for all incident light rays. This model groups the catadioptric cameras used to acquire the whole scene using, for example, a single photosensitive sensor array and a curved mirror. Panoramas created by a rotating camera around its nodal point, or its projection center assuming a pinhole camera model, also satisfy the SVP model. These panoramas are created by acquiring planar images to be mosaicked or by scanning the scene column-wise, i.e., using line-sensor cameras and turning platforms. Examples of SVP acquisition are illustrated in Fig. 2(a) and 2(b).

Fig. 2

Examples of omnidirectional image acquisition: (a) catadioptric cameras based on parabolic or hyperbolic mirrors produce SVP panoramas,² (b) rotating a camera about its nodal point to acquire multiple perspective projections with a common projection center also produces SVP panoramas, while (c) rotating an off-centered camera to acquire image patches, around a point different than its nodal point, produces non-SVP panoramas, as well as (d) multi-sensor cameras, such as the Ladybug2 panoramic camera,³ which also produce non-SVP panoramas.

In the case of a non-SVP model, the panoramic image is rendered using a centrally symmetric set of projection centers which are not spatially collocated. Cameras based on the non-SVP paradigm are more common than those based on an SVP model because the physical dimension of multiple camera configurations prevents sampling the scene from a single viewpoint. A way around this problem is using planar mirrors to reposition the projection centers closer to each other, approximating an SVP configuration. In the context of the problem studied in this paper, stereoscopic panoramas are by definition non-SVP panoramas since the scene is imaged from two distinct viewpoints (left- and right-eye viewpoints) for any possible gazing direction. Examples of non-SVP cameras are shown in Fig. 2(c) and 2(d).

2.2.

Omnistereoscopic Acquisition Models

The different strategies to acquire the necessary visual information to produce stereoscopic panoramas (in a cylindrical format) can be summarized into a limited number of acquisition models. We propose to reduce the classification to four models constrained to acquire stereoscopic panoramic imagery for human viewing. Hence, these models are conceived to represent the acquisition of two images of the same scene from two distinct viewpoints with horizontal parallax. Each of these models represents the stereoscopic acquisition of image pairs for multiple gazing directions in azimuth, and for a limited field of view (FOV) in elevation. All the cameras and acquisition techniques reviewed in this paper can be modeled by one of these four models.

The proposed models are suitable to describe the sequential acquisition of visual information toward the rendering of stereoscopic panoramas. A few of the proposed acquisition models are limited to sequential sampling since inherent self-occlusion problems prevent them from being implemented using multiple sensors. But some of the proposed cases can also model the simultaneous scene acquisition, i.e., using multiple sensor configurations or other omnidirectional camera systems. The simultaneous acquisition case is of particular interest in the context of the problem studied in this paper.

The first stereoscopic acquisition model is the central stereoscopic rig, which is illustrated in Fig. 3(a). In this case, two coplanar cameras separated by a baseline $b$ determine a viewing circle concentric with the geometric center of the camera arrangement. The viewing circle is the virtual circle determined by the trajectory of both cameras while panning all azimuthal angles ($0\mathrm{deg}<\theta \le 360\mathrm{deg}$) around $O$. This model for omnistereoscopic rendering has been widely used in the literature over the last decade to represent a stereoscopic rig panning the scene in azimuth.⁴ This model is suitable to represent the sequential acquisition of partially overlapped stereoscopic image pairs with respect to a common center.⁵ In all of the four acquisition models, a single- or dual-camera system samples the scene for different $\theta$ on a plane $X^{\prime }{Z}^{\prime }$ parallel to the reference floor $XZ$ as illustrated in the binocular viewing model of Fig. 1. Finally, this model can also represent a widely used technique based on extracting two columns, corresponding to the left- and right-eye perspective, from the sequence of planar images acquired using single camera rotated off-center.^6,7 However, due to self-occlusion between cameras, this model cannot be applied in a parallel acquisition configuration.

Fig. 3

Omnistereoscopic acquisition models using multiple-camera configurations: (a) central stereoscopic rig, (b) lateral stereoscopic rig, (c) lateral-radial stereoscopic rig, and (d) off-centered stereoscopic rig.

The lateral stereoscopic rig model is shown in Fig. 3(b). This model represents a viewing circle centered at the projection center $O$ of one of the two cameras. The off-centered camera (stereoscopic counterpart) describes a circle of radius equal to the stereo baseline $r_c=b$ while rotating around the center $O$. In this model, one camera is used to produce an SVP panorama centered at $O$ (nodal point of the central camera), while the second camera is used to estimate the scene’s depth by acquiring stereoscopic counterparts of the images acquired by the central camera.^8–11 This method enables horizontal disparity estimation and the extraction of occlusion information to be used in the rendering. For similar reasons as in the previous acquisition model, this acquisition model cannot be used in a parallel acquisition scheme due to self-occlusion between cameras.

The lateral-radial stereoscopic rig model, which is shown in Fig. 3(c), can be derived from the lateral stereoscopic rig model presented above, by adding a radial distance ${\mathbf{r}}_{\mathbf{c}}$ between the symmetry center $O$ and the nodal point of one of the cameras (central camera in the previous model). This is a more general model where the nodal points in a multiple-sensor arrangement cannot be concentric due to the physical dimension of each camera.¹² The lateral-radial stereoscopic rig model can also represent a stereoscopic rig rotated off-center, where one camera is radially aligned with the center $O$, while the second camera is horizontally displaced $b$ to capture another snapshot with horizontal parallax. This model can represent a parallel acquisition scheme, i.e., a multiple-sensor arrangement.

The off-centered stereoscopic rig models a stereoscopic rig located at a radial distance ${\mathbf{r}}_{\mathbf{c}}$ from the geometrical center $O$ as depicted in Fig. 3(d). This model is suited for camera configurations where multiple cameras, usually a large number of them, are radially located with respect to a center $O$. These cameras, when taken in pairs, define a series of radially distributed stereoscopic rigs. The partially overlapping FOV between even (or odd) cameras can be used to mosaic vertical slices of the scene, rendering a stereoscopic pair of panoramas.¹³ Multicamera configurations have been proposed¹⁴ using $N$ ($N\ge 5$) radially distributed stereoscopic rigs. These configurations are based on acquiring a number of partially overlapped stereoscopic images of the scene. This model of acquisition can also represent a parallel acquisition of multiple stereoscopic snapshots of the scene.

2.3.

Comparing Different Camera Configurations

Several omnistereoscopic acquisition and rendering techniques have been proposed over the last decade. Most of them are not suitable for acquiring dynamic scenes omnistereoscopically, but some configurations satisfy this constraint. Unfortunately, the pros and cons of the panoramic cameras suitable for the task are still open to research. In order to understand the limitations of the different camera configurations, we simulated some basic characteristics of the four configurations presented in the previous section.

One fundamental aspect to consider is the continuity of the horizontal disparity between partially overlapped stereoscopic snapshots. This is particularly important when the rendering is based on mosaicking. In Fig. 4, we compared the relative variation in the minimum distance to maintain continuity in the horizontal disparity between mosaics. The idea is to find the minimum distance to the scene to have subpixel variations in the horizontal disparity between adjacent image samples. Our simulations were based on an APS-C sensor size ($22\mathrm{mm}\times 14.8\mathrm{mm}$) of 10.1 megapixels. The case presented here, which is shown as an example only, corresponds to one particular combination of baseline $b=65\text{mm}$ and lenses’ FOV $45°$ for the four camera models. The simulation result shows a reduction in the minimum distance for stereoscopic rendering achievable for the all the acquisition models, compared against Model 1, as a function of the blending position in each image. In this particular example, for the camera Models 2 and 3, the relative minimum distance increased more than in other camera models when the blending threshold is above 12% of the image width $W_h$, measured from the edge of each image to mosaic.

Fig. 4

The acquisition models are contrasted against the central stereoscopic rig (model 1) showing the relative variation of the minimum distance to the scene to achieve horizontal disparity continuity among neighbor images: the compared configurations are lateral stereoscopic rig (model 2), the lateral-radial stereoscopic rig for $r_c=b$ (model 3), and the off-centered stereoscopic rig for $r_c=b$ (model 4).

Also important in the camera design is the minimum distance to the scene to have an object imaged by adjacent stereoscopic sensor pairs; in other words, how far the stereoscopic FOV is located with respect to the panoramic camera. This distance depends on the multiple cameras’ geometric configuration, the FOV of each camera, and the stereoscopic baseline. In another example, we contrasted the minimum distance for stereoscopic rendering using the same baseline and changing the lenses’ FOV only. The results presented in Fig. 5 correspond to the camera Model 1 for a fixed baseline length $B=35\text{mm}$ and three FOV cases. The simulation results show that the minimum acceptable distance to maintain stereoscopic continuity between mosaics as a function of the blending position in each image. As in the previous case, the blending position is expressed as a horizontal length, measured from the edge of the image as a percentage of the image width.

Fig. 5

Minimum distance to the scene for the central stereoscopic rig (Model 1) for $b=35\mathrm{mm}$ and different lenses’ FOV.

2.4.

Problem

The main problem of the omnistereoscopic image acquisition for human viewing is how to sample the complete visual field at once, from two viewpoints with horizontal parallax. Furthermore, if using multiple cameras to do this, how to avoid the self-occlusion between cameras and how to minimize the problems introduced by sampling the scene from close but different viewpoints.

The self-occlusion problem is common to all the conceptualizations of panoramic photography, which must be considered when a single or multiple cameras are used to sample the scene omnidirectionally. If the image sampling is sequential, self-occlusion can be avoided. However, the acquisition of dynamic scenes exacerbates the restrictions since all the information to produce omnistereoscopic images has to be acquired at once. The parallax arising from sampling the scene from different viewpoints is another problem common in panoramic photography. The problem gets more complicated when the simultaneous acquisition of stereoscopic images from different viewpoints enters into the equation.

One possible solution to the problem is to acquire multiple stereoscopic snapshots of the scene simultaneously. In this case, the geometric configuration of the multisensor device must be carefully designed to avoid self-occlusion that occurs when one camera lies in the FOV of another. Alternatively, another possible solution is using diffractive optics to obtain two views of the scene with horizontal parallax, and doing so omnidirectionally. In this case, the image formation for this type of diffractive lens has to be modeled and the capabilities of such a camera have to be assessed.

A camera under these constraints should be able to acquire an omnidirectional binocular snapshot of the whole scene. The information captured by this camera should be sufficient to render two non-SVP panoramic views corresponding to the left and right eyes or, more generally, for stereoscopic renditions of the scene in any arbitrary gazing direction.

2.5.

Panoramic Acquisition: Cameras and Methods

The omnistereoscopic technologies reviewed in this paper were classified into four families based on their image acquisition strategies and/or their constructive characteristics.

This classification in families is independent of the four models of omnistereoscopic image acquisition presented in Sec. 2.2. Nevertheless, each omnistereoscopic technology representative of these four omnistereoscopic families can be modeled using one of the four acquisition models introduced above. The catadioptric cameras based on vertical parallax are the only exception to this rule, since all the presented acquisition models are based on horizontal stereo.

In the following section, the pros and cons of each family are studied individually, distinguishing those cameras whose characteristics can be adapted to an omnistereoscopic configuration suitable for acquiring dynamic scenes.

3.1.

Catadioptric Omnistereoscopic Cameras

The development of omnistereoscopic catadioptric cameras has paralleled the development of general (monoscopic) catadioptric sensors. It should be mentioned that omnistereoscopic catadioptric cameras were originally intended for the real-time estimation of depth maps. In other words, these omnistereoscopic approaches were not intended to produce omnistereoscopic imagery for human viewing, but they were motivated by applications such as robot navigation and 3-D scene reconstruction. One important remark is necessary here: the omnistereoscopic cameras based on a catadioptric configuration with vertical parallax presented in this section are not modeled by any of the acquisition models presented in Sec. 2.2 since the omnistereoscopic acquisition classification has been constrained to human-viewable omnistereoscopic imagery, i.e., binocular stereo with horizontal parallax.

One of the earlier examples of this technology was an SVP catadioptric camera proposed by Southwell et al.²³ in 1996. This omnistereoscopic catadioptric camera is based on a coaxial, dual-lobe parabolic mirror, and its main application was to generate omnidirectional depth maps of the terrain. A depiction of this camera appears in Fig. 8(a).

Fig. 8

Omnistereoscopic catadioptric examples: (a) the camera proposed by Southwell et al.²³ in 1996 using dual-lobe mirror and (b) an early SVP catadioptric omnistereoscopic camera proposed by Gluckman²⁴ in 1998: this configuration uses two coaxial catadioptric panoramic cameras with a large vertical baseline to acquire two panoramic views of the scene with vertical parallax $b$; the 3-D scene structure is estimated from the vertical disparity arising between matched feature points in each panorama.

Another configuration was proposed by Gluckman et al.²⁴ in 1998. Their configuration is based on using two coaxial catadioptric cameras whose vertical baseline helps to acquire an omnistereoscopic pair of images. This camera, illustrated in Fig. 8(b), enables the estimation of the 3-D location of a scene point $P$ in space, matching pairs of feature points $(p,p^{\prime })$ in the panoramic views from each camera. The larger the vertical baseline $b$ used, the better the accuracy of depth estimation achieved. A theoretical model of the image formation in a dual-mirror, axially symmetrical, catadioptric sensor was proposed by Stürzl et al.²⁵

The coaxial catadioptric camera has been a popular method to estimate omnidirectional depth over the past two decades due to its simplicity and hardware economy. Unfortunately, it is not suitable to produce a satisfactory omnistereoscopic rendition of the scene capable to stimulate human stereopsis. The binocular visual process is based on fusing two views of the scene obtained from horizontally displaced viewpoints (horizontal parallax). This camera provides two views of the scene in every direction, but based on vertical parallax. Although the omnidirectional depth can be estimated using this information, it cannot be used to produce satisfactory horizontal stereoscopic views. For instance, when using a two-dimensional (2-D) to 3-D conversion, the gaps in image coming from occluded areas need to be filled, e.g., using texture synthesis.²⁶ One of the problems of the vertically coaxial method is that the visual information regarding occluded areas is not acquired.

A similar omnistereoscopic camera based on the catadioptric principle was proposed by Kawanishi et al.²⁷ in 1998. Their camera consists of two non-SVP catadioptric cameras in vertical coaxial configuration as shown in Fig. 9(a). Each catadioptric camera consists of six planar mirrors, each of which reflects a partial view of the scene over a video camera. This configuration produces 12 video streams covering 360 deg in azimuth. Each camera in the top of the arrangement is paired with the camera located directly below, i.e., the cameras $n$ and $n^{\prime }$ form a stereoscopic camera pair, whose vertical baseline is $b$, as illustrated in Fig. 9(b). Similar to Gluckman’s camera,²⁴ the vertical parallax $b$ between camera pairs enables the panoramic estimation of the scene’s depth but does not provide the means to render viewable stereoscopic images. In a follow-up of this design, Shimamura et al.²⁸ built a working prototype based on the Kawanishi et al. design capable of producing panoramic depth maps.

Fig. 9

Omnistereoscopic camera configuration based on coaxial planar mirrors: (a) configuration based on Kawanishi et al.’s idea²⁷ and (b) virtual location of each camera’s projection center and vertical baseline $b$.

Spacek²⁹ relaxed the non-SVP condition using two conic mirrors, instead of pyramidal mirrors, coaxially aligned with cameras. This configuration was conceived to estimate distances based on vertical disparities. The author reported benefits over other profiles in using conical mirrors in terms of the uniformity of the resolution density. However, this type of profile introduces out-of-focus blurring in some regions of the orthographic image because the optical focus is not uniformly located as in the case of hyperbolic and parabolic mirrors.

A recent interesting development is due to researchers at the Fraunhofer Heinrich Hertz Institute,³⁰ who are currently working on a prototype of an omnistereoscopic high-definition television (HDTV) camera based on a catadioptric design. Conceived for omnistereoscopic 3-D TV broadcasting, this setup uses six stereoscopic camera rigs, each of which is associated with a planar mirror. Each mirror reflects a partial view of the scene on a camera pair, for a total of 12 HDTV cameras.^31,32 These video streams can be mosaicked into a full omnistereoscopic video, or into free-panning 3-D TV signal. The concept of this camera is presented in Fig. 10. The creators of this camera have reported difficulties when mosaicking partially overlapping stereoscopic frames due to the large parallax between adjacent projection centers.³³ Part of the problem resides, as in other star-like configurations, in the excessive parallax introduced by the stereoscopic rigs where both cameras are laterally displaced with respect to each other. The minimum distance to objects in the scene for correct rendering is affected by the large intercamera parallax, reducing the stereoscopic usability in foreground regions of the scene. This camera configuration can be represented by the off-centered stereoscopic rig acquisition model [Fig. 3(d)], where six stereoscopic camera rigs, equally distributed at a distance ${\mathbf{r}}_{\mathbf{c}}$ from the geometric center $O$, simultaneously capture six partially overlapped, stereoscopic video signals of the scene.

Fig. 10

Omnistereoscopic video camera developed at the Fraunhofer Heinrich-Hertz Institute: (a) each planar mirror face is associated with a stereoscopic pair of cameras, (b) locations the cameras as seen reflected on the planar mirrors.

Peleg et al.³⁴ proposed a different catadioptric camera configuration capable of acquiring horizontal binocular stereo omnidirectionally and in real time. This catadioptric system uses a complex spiral lenticular lens and an optical prism to acquire a complete omnistereoscopic image in real time. This configuration deflects incoming light rays as if the scene were acquired simultaneously from multiple perspective points located on a virtual viewing circle (Sec. 2.2). This camera can be modeled by the central stereoscopic rig acquisition model [Fig. 3(a)], where a large number of stereoscopic image vertical stripes (central columns of left and right images) are simultaneously sampled and mosaicked to create complete left and right cylindrical panoramas in real time. The proposed lenticular arrangement is shown in Fig. 11(a), which illustrates the acquisition of a one eye (left or right) panoramic view. The idea of using a Fresnel-like diffractive system with thousands of lenses to capture both omnistereoscopic views simultaneously could, in theory, produce an omnistereoscopic video in real time. The lenticular arrangement can be built around an SVP panoramic camera as shown in Fig. 11(b). The authors have proposed using a beam splitter and the described lenticular system to acquire both viewpoints of a stereo projection simultaneously as illustrated in Fig. 11(c).

Fig. 11

Peleg et al.’s³⁴ proposal for a real-time omnistereoscopic camera based on a catadioptric principle: (a) a Fresnel-like lenticular lens arrangement diffracts the light over a viewing circle, (b) a catadioptric scheme with a cylindrical diffractive material composed of vertical stripes of the proposed Fresnel lens to capture one (left- or right-view) panorama, and (c) using an optical beam splitter, e.g., a prism, and combining diffraction lenses for left and right view in the same cylindrical surface, both (left- and right-eye) views can be captured simultaneously.

This camera could be a solution to the problem of omnistereoscopic image acquisition of dynamic scenes. Furthermore, Peleg, Ben-Ezra, and Pritch were granted a patent³⁵ for this camera in 2004, but no prototype has yet been built or licensed to the best of our knowledge. This commercialization lag must not be taken as a proof of the inadequacy of the idea; e.g., more than 70,000 patents were granted annually in the United States by the turn of the century and only a very few of them were commercially developed.³⁶ As a matter of fact, a variation of the lenticular lens, although not an omnistereoscopic application, has been licensed for the production of 3-D billboards.³⁷ Peleg et al. proposed a geometrical model of their elliptic lens;³⁸ however, there still are aspects of the image formation for this camera that have not been extensively studied. More importantly, the capabilities of such an optical approach to produce high-quality omnistereoscopic imagery has not been demonstrated.

It is important to remark that a cylindrical stereoscopic panorama produces a correct binocular experience only in the center of each rectangular view extracted from these left and right panoramas. The peripheral image region, outside the image center, produces a distorted binocular perception. This effect was mentioned by Bourke^39,40 while addressing the problem of semi-spherical omnistereo to display in dome surfaces. Although this is a noticeable effect, the user tends to focus on a region of interest (ROI) at the center of the image, where the binocular perception is correct, reducing the likelihood of uncomfortable effects that could lead to eye strain. Furthermore, if a cylindrical omnistereoscopic image created with this acquisition method was projected in a cylindrical surface around the user (located at the center), a correct binocular depth would be experienced by looking in any direction around the user, as long as the zero parallax distance (eyes vergence) coincides with the distance to the cylindrical screen.⁴¹

Other configurations based on catadioptric cameras that appeared in the last decade mainly focused on the problem of omnidirectional stereo reconstruction^42,43 following the idea of coaxial catadioptric stereo, which makes them inadequate to produce omnistereoscopic imagery suitable for binocular human viewing.

3.2.

Pros and Cons of the Catadioptric Omnistereoscopic Cameras

The most evident advantage of an SVP catadioptric camera is its simplicity: a single camera and dioptric system can sample the scene’s visual field, in addition to estimating the panoramic depth, in a single snapshot. The SVP approach avoids stitching problems and imperfections that arise by parallax between multiple projection centers in non-SVP cameras. However, focusing the light uniformly on a planar image sensor after reflecting light in a nonplanar mirror is problematic. Not all the rays are parallel after reflection in a parabolic surface, nor are they perfectly reflected toward the convex focal point in a hyperbolic mirror. In practice, image blurring as a function of the radial distance from the center of the image¹⁷ is difficult to avoid. This problem can be reduced, although not completely, by a careful design of the catoptric system (mirror) and its dioptric (focusing) counterpart. The high-resolution CCD sensors available nowadays can help to reduce problems while resampling the acquired orthographic projection of the scene into a canonical panoramic surface. Furthermore, a catadioptric camera has advantages with respect to other mirror-less systems in terms of reducing chromatic aberrations present in most aspheric lenses, i.e., fisheye lenses. Offsetting the advantages of catadioptric cameras, there are still inherent problems in using this camera to render an omnistereoscopic scene for binocular viewing.

Although catadioptric omnistereoscopic configurations using vertically coaxial mirrors are undoubtedly an elegant method for acquiring omnidirectional depth maps in real time, the camera configurations are unable to handle occlusions in the scene. A binocularly correct stereoscopic view can be synthesized with one panoramic image plus a dense horizontal disparity information of the scene. The disparity information can be used to generate a pixel-wise horizontal displacement map. Applying this map to horizontally displace pixels (or regions) in the image can produce a correct illusion of depth. The necessary information to produce this omnistereoscopic image can be acquired using an omnistereoscopic catadioptric configuration: a correct panorama of the scene and an omnidirectional depth map. Unfortunately, the information to fill image gaps (occluded areas) cannot directly be obtained using a camera configuration based only on vertical parallax. But, there are still suboptimal solutions coming from the field of 2-D to 3-D conversion that can be applied. For instance, pixels can be copied from the adjacent regions of the missing areas to fill these gaps. A much better alternative will be to simultaneously acquire a different view, e.g., a horizontal stereoscopic pair for each gazing direction. The parallax view information can be used to fill-in the occluded areas using a texture inpainting technique.⁴⁴

A good candidate to produce directly viewable omnistereoscopic imagery, capable of satisfing the real-time acquisition of dynamic events as well, is the optical diffractive approach proposed by Peleg et al.,^35,45 as illustrated in Fig. 11. However, the design and implementation of such a lenticular arrangement is challenging. Similar to Peleg’s approach, there has been at least one other recent proposal, which is also based on lenticular arrays to multiplex two horizontal binocular views of the scene, but using non-SVP configurations,⁴⁶ which will be presented in Sec. 6. Besides the lack of commercial interest for an optical-based approach, an alternative solution, based on off-the-shelf lenses and camera sensors, can provide a satisfactory solution to the problem with reduced hardware complexity.

Another good candidate is the panoramic 3-D TV camera developed at the Fraunhofer Heinrich-Hertz Institute. This omnistereoscopic camera, which is illustrated in Fig. 10, uses multiple off-the-shelf HDTV cameras and mirrors to produce real-time stereoscopic videos with broadcasting quality. The downfall of this camera is its size³¹ which makes difficult the mosaicking of individual video streams. One possible solution for the parallax problem would be reducing the size of the cameras, e.g., using custom-made HDTV cameras. Additional improvement may be achieved by using a different camera distribution to enable registering stereoscopic images of scene elements closer to the camera.

4.1.

Omnistereo Based on Sequential Acquisition Models

Perhaps one of the most illustrious applications of omnistereoscopic sequential acquisition was integrated to the Mars Pathfinder rover⁴ in the late 1990s. The camera can be modeled by the central stereoscopic rig method illustrated in Fig. 3(a). It was designed to provide a variety of telemetric measurements beyond producing omnistereoscopic imagery; actually, producing stereoscopic images suitable for human stereopsis was not the primary goal of this camera. Two cameras were mounted on a retractable mast and were rotated around the midpoint between each camera’s nodal point. The pair of cameras, whose resolution was modest as is expected for an interplanetary probe of that era ($512\times 256$), were toed in 0.03 rad, defining a fixation point (zero parallax distance) at 2.5 m from the rotation center. The stereoscopic panoramas produced by this camera received much attention in the news.^49–51 Although these omnistereoscopic images are not impressive in terms of quality of the binocular stereo, they constitute an important precedent for the marketing value of realistic immersive imagery to promote a planetary mission to the layperson.⁵²

Other authors reported variations on the rotating stereoscopic rig method that are worth mentioning. For instance, Romain et al.⁵³ proposed a rotating platform with two degrees of freedom, which can rotate in azimuth and elevation to produce spherical omnistereoscopic images. More recently, Ainsworth et al.⁵⁴ revisited the method of a rotating stereoscopic rig to minimize (not to eliminate as the authors stated) stitching problems. They reported their method to create stereoscopic panoramas⁵⁵ based on mosaicking partially overlapped scene's images, which they tested using an off-the-shelf digital camera (Panasonic Lumix) and a commercial rotating platform (GigaPan EPIC Pro Robotic Controller).

Huang et al.⁸ proposed a rotating camera to acquire stereoscopic images of scenes, which were aligned and mosaicked to render omnistereoscopic images. Their idea, originally published in 1996, is illustrated in Fig. 12(a), which shows a central camera that is rotated around its nodal point and a second camera that is rotated off-axis, providing a parallax view of the scene. Their method corresponds to the lateral stereoscopic rig model [Fig. 3(b)], which produces non-SVP stereoscopic panoramas, where the stereo budget can be selected by choosing the baseline $r_c=b$ of the rotating stereo rig. An interesting idea is to acquire stereoscopic images of the scene using a rotating stereoscopic rig, where one camera (central) is in the optimum position to minimize stitching errors (nodal point), while the second camera captures a second view of the scene with horizontal parallax. The central camera produces an SVP panorama while the second camera can be used to render a binocular polycentric panorama. Because of camera self-occlusion, this idea is difficult to implement using multiple cameras, although a multicamera configuration using planar mirrors can approximate an SVP for a central camera and acquire a horizontal parallax view simultaneously. There have not been proposals based on this catadioptric design to our best knowledge.

Fig. 12

Omnistereoscopic methods based on sequential acquisition of partially overlapped images: (a) method proposed by Huang et al. in 1996 to generate a correct panorama (central camera) and accessory information to estimate a horizontally parallax view (lateral camera) and (b) the acquisition strategy proposed by Yamada et al., which is similar to Huang’s method, is based on acquiring images to produce an SVP panorama (central camera) and, in this case, estimating the panoramic depth map based on a large-baseline stereoscopic pair of images (left and right cameras).

Using a similar approach, Yamada et al.^9–11 proposed the triple camera system shown in Fig. 12(b). Similar to Huang et al.’s method, the central camera produces an SVP panorama while the two satellite cameras help to estimate a panoramic disparity map using a large baseline. This sequential acquisition technique can be seen as a modification of the central stereoscopic rig [Fig. 3(a)], used only to estimate depth by exploiting the larger baseline ($b=2·r_c$) between the satellite cameras, and adding a central camera to produce an SVP panorama as in the lateral stereoscopic rig model. Again, self-occlusion between cameras makes it difficult to parallelize Yamada’s approach in a simultaneous acquisition scheme. But a catadioptric scheme to acquire an SVP and a multisensor scheme to simultaneously acquire a number of partially overlapped stereoscopic snapshots of the scene is an interesting design challenge and a suitable approach for dynamic scene omnistereoscopic acquisition. No camera following the suggested approach has been proposed to our best knowledge.

A variation of the idea of rotating sensors was proposed by Chan and Clarke⁵⁶ who devised a camera where a mirror is rotated while a single sensor sequentially captures binocular stereoscopic images of the scene. Their idea was inspired from endoscopic applications where a single probe has to be inserted in a biological cavity or in an archeological site. The principle behind the idea is similar to other proposals in terms of a sequential acquisition. For instance, carefully selecting the planar mirror and camera locations, the central stereoscopic rig acquisition scheme of Fig. 3(a) can be implemented with this technique.

Ishiguro et al.⁶ proposed in 1992 a method to create omnistereoscopic imagery based on a single rotating camera, but for robot navigation instead of human visualization. This method corresponds to the central stereoscopic rig acquisition model illustrated in Fig. 3(a). Peleg and Ben-Enzra⁷ rediscovered this method in the late 1990s, but tailored the idea with human visualization in mind. This method has become one of the most popular sequential techniques to create high-quality omnistereoscopic imagery given its hardware economy and simplicity. The principle is based on a single camera rotating around a point behind its projection center, as depicted in Fig. 13(a). In one complete rotation, the camera captures a number of images that are used to extract left (${\mathrm{im}}_L$) and right (${\mathrm{im}}_R$) columns. These two image columns correspond to the back-projection of the image’s vertical stripes defined by intersecting the rays passing through the camera projection center $O$ and the points $O_L$ and $O_R$. The ray tracing concept is depicted in Fig. 13(b). These columns are then mosaicked producing left- and right-eye panoramic views. The end result is equivalent to the central stereoscopic rig model [Fig. 3(a)], used to acquire left and right views column-wise. There have been proposals for omnistereoscopic cameras that can be directly traced to parallelization of a single rotating camera.^57,58,59

Fig. 13

Rotating method to produce omnistereoscopic imagery based on a single rotating camera: (a) a single camera is rotated from an off-centered location and (b) two projections corresponding to left- and right-eye projections can be defined intersecting the rays passing through the camera’s projection center $O$ and the points $O_L$ and $O_R$ defined over a virtual viewing circle.

The method has the advantage of defining a virtual baseline $(b=2·r_c)$ that can be varied according to the stereoscopic budget desired for the scene by changing the relative distance between left- and right-eye vertical stripes extracted from the successively acquired images.⁶⁰ Changing the distance has the effect of changing the viewing circle diameter. More recently, Wang et al. proposed an interactive method to adjust the disparity in panoramas created using the single off-centered rotating camera⁶¹ based on the interactive selection of objects in the scene.

Examples of 3-D (not stereoscopic) images created using the central stereoscopic rig method for monoscopic panoramas and omnistereoscopic images can be visited online.^62,63 Additionally, several patents have been granted to Peleg et al. for acquiring, disparity adjusting, and displaying stereoscopic panoramas using the single-camera method,^64–66 one of which has been licensed to HumanEyes,³⁷ not to create omnistereoscopic imagery but to create multiview 3-D billboards.

A different approach was proposed by Hongfei et al.⁵ who used a single digital single-lens reflex camera to successively acquire 12 images of a static scene. First, three stereoscopic image pairs (six images in cd by placing the camera in the positions labeled $iL$ and $iR$ [for $i=(1,2,3)$] as indicated in Fig. 14(a). After that, the camera is rotated 180 deg and six new stereoscopic images are successively acquired following the pattern illustrated in Fig. 14(b). This acquisition scheme corresponds to the central stereoscopic rig model illustrated in Fig 3(a). Although Hongfei et al.’s method is not suitable for dynamic scenes, a parallelized version of the same idea was already proposed by Baker et al.¹⁴ as a patent application in 2003.

Fig. 14

Method based on rotating a single camera to capture six stereoscopic images by positioning the camera in 12 different locations and orientations: (a) first, the stereoscopic pairs $[(1_L,1_R),(2_L,2_R),(3_L,3_R)]$ are acquired one-by-one rotating and positioning the camera to the corresponding locations. (b) Finally, the camera is rotated 180 deg around its nodal point and the pairs $[(4_L,4_R),(5_L,5_R),(6_L,6_R)]$ are sequentially acquired.

Among the panoramic sequential methods, those based on using line-sensor cameras deserve a particular mention. Cylindrical omnistereoscopic imagery obtained by this method produces geometrically correct binocular views of the scene at the center of the image, while the depth perception is distorted in the periphery of the image.⁶⁷ This viewing paradigm is valid for cylindrical panoramas projected in a cylindrical display, in both monoscopic and stereoscopic panoramas, in part because it does not have to deal with parallax ghosting while blending images. Sequential line scanning is based on mosaicking image columns a single pixel wide and therefore it produces high-quality stereoscopic panoramas. However, this virtue is offset by their lengthy acquisition time, which is common to all sequential-scanning methods. The line-scanning methods to acquire omnistereoscopic imagery can be modeled by the central stereoscopic rig acquisition method [Fig. 3(a)].

High-quality omnistereoscopic images, in cylindrical format, can be produced using line-scanning sensors. This sequential method has been studied over the first decade of the 2000s,⁶⁸ thanks, in part, to the commercial availability of line-scanning cameras.⁶⁹ As their name indicates, these cameras acquire the scene column-wise. An omnistereoscopic view of the scene can be acquired by rotating the line-scanning sensor off-center at radius ${\mathbf{r}}_{\mathbf{c}}$ to acquire independently left ($I_R$) and right ($I_L$) cylindrical panoramas column-by-column. The acquisition model corresponds to the central stereoscopic rig method illustrated in Fig. 3(a), where a line-scanning sensor is in the position of each of the cameras for one complete rotation, acquiring successively each eye viewpoint after two complete rotations. Several authors have contributed to the understanding and modeling of omnistereoscopic imagery using line sensors, covering the line-scanning camera calibration,^68,70 a geometrical model for polycentric panoramas using this acquisition strategy,⁷¹ and the omnistereoscopic image acquisition.^72,73–75 Although the literature on line sensors is extensive and insightful, this approach cannot be adapted to acquire dynamic scenes.

Hybrid approaches have been proposed that use a laser range finder and rotating sensors to provide a high-resolution depth mapping of the scene. For instance, Jiang and Lu⁷⁶ proposed a method that combines an off-center CCD camera with a laser range sensor, which together produced a monoscopic panorama and its dense depth map. Their approach is shown in Fig. 15. Once again, the problem addressed was the reconstruction of a 3-D scene and not the production of a binocular omnistereoscopic imagery. Besides its sequential conception, this idea cannot be used for an omnistereoscopic 2-D to 3-D conversion since occlusion information is not collected during the sequential acquisition. Similar limitations in handling occlusions arise in a recent proposal by Barnes,⁷⁷ which combines ground-based LIDAR and monoscopic panoramas.

Fig. 15

Combined camera and laser range sensor in a rotating platform for sequential acquisition: an SVP panorama plus a panoramic depth map can be obtained for static scenes using this acquisition method.⁷⁶

The rotating sensor for omnistereoscopic imagery has appeared with some variations over the last decade.^67,78 A good summary of methods to create omnistereoscopic images based on rotating sensors was published in 2006 by Burke.⁷⁹

4.2.

Pros and Cons of the Sequential Acquisition

Sequential acquisition strategies are an interesting starting point to devise multicamera configurations for real-time omnistereoscopic acquisition. The configuration proposed by Huang and Hung⁸ is interesting since it presents a solution to reduce the parallax-induced errors while stitching multiple images. This method can be parallelized to simultaneously acquire a number of image patches and their corresponding stereoscopic counterparts in a non-SVP scheme. The sequential method of Peleg and Ben-Enzra using a single off-centered camera⁷ is interesting for reducing the hardware involved, but it is difficult to parallelize since it would require a prohibitively large number of cameras to prevent blending artifacts while mosaicking. In that case, the mosaicked image columns have to be constrained to a few pixels wide, which implies taking hundreds of snapshots. A multiple-camera configuration that attempts to take this number of simultaneous pictures is impractical, but, as will be shown in Sec. 6, some multiple-camera configurations that do this parallel acquisition have been proposed.

The large overhead when acquiring hundreds of snapshots to just use two narrow slices is partially compensated by line cameras, which scan the scene column-wise. Unfortunately, the large acquisition time of line cameras limits them to controlled static environments, e.g., mostly indoor scenarios.

However, it is possible to devise improvements in acquisition speed of omnistereoscopic images using line sensors. For instance, using multiple stereoscopic line sensors, the acquisition speed can be increased linearly with the number of stereoscopic sets. This approach would enable the simultaneous capture of nonoverlapping stereoscopic images of the scene, which can then be mosaicked to create a full omnistereoscopic image in a fraction of time that a single sensor can achieve. Unfortunately, a rotating camera system with such characteristics is still commercially unavailable and, if it were available, it would not be suitable to acquire dynamic scenes.

5.1.

Pros and Cons of Using Panoramic Sources

Although the sequential methods based on panoramic sources are not suitable to acquire dynamic scenes, some of them can be adapted to multisensor configurations using wide-angle cameras. This is the case of clusters of panoramas where the distance and spatial location between projection centers are precisely known. An example of this parallelization in the acquisition is the camera proposed by Baker et al.¹⁴ using four panoramic sources and the sequential approach proposed by Vanijja et al.²⁰ using the same square pattern. Similar parallelization can be applied to the authors’ own proposal of using triangular clusters of coplanar cylindrical panoramas. The main cons of using panoramic sources to extract wide-angle stereoscopic views of the scene are the natural self-occlusion between panoramas in parallel acquisition approach that leads to large data overhead.

6.1.

Pros and Cons of Multicamera Configurations

The parallelization of the central stereoscopic rig acquisition method⁵⁷ would be an interesting option, especially given the trend in miniaturization of high-quality cameras. The reason for being skeptical about this approach is simple: the narrower the images’ vertical stripes used to extract two parallax views of scene, the better the quality of mosaicking, but the necessary number of cameras to be able to extract narrow image columns is too large to be practical.

A radial arrangement to capture partially overlapped stereoscopic images using reasonably large baselines implies a careful design of the camera geometry. Multiple-camera configurations using this strategy have been proposed recently.^{12,14,46,93,94} However promising, there are still unsolved issues regarding the cameras’ parallax effect in the rendering of intermediate views between samples, e.g., stitching panning directions, and maintaining the disparity continuity. Moreover, the effect of the geometric distribution of cameras on the minimum distance to objects on the scene for a correct stereoscopic viewing has not been properly studied.

Another point to be considered when mosaicking wide-angle stereoscopic samples is the distortion incurred in creating a cylindrical stereoscopic panorama by mosaicking a limited number of partially overlapped stereoscopic snapshots.

The conclusions of this background research in terms of the potential of each panoramic camera to acquire dynamic scenes omnistereoscopically are summarized in Table 1.

Table 1

Comparison between different omnistereoscopic methods.