1.

INTRODUCTION

Virtual Reality, Mixed Reality, and Augmented Reality (collectively here “VR”) headsets are of increasing importance in our daily lives. They promise enhanced consumer experiences and more efficient workplaces through directly assisting operators and engineers. Virtual reality headsets typically refer to systems which generate a new threedimensional world independent of one’s physical surroundings, where mixed and augmented realities imply a mixed image of user surroundings and rendered graphics. For the latter experiences, the interconnect to the physical world may be electronic, where onboard cameras convert the image of the external world into part of the rendered graphics for the display, or optical, where rendered graphics are channeled into passive optics. Purely emissive display technologies for head-mounted display (HMD) applications, including the one presented, are suitable for any or all of virtual reality, augmented reality, and mixed reality headsets.

Lenslet arrays (also called microlens arrays, or, in one dimension, lenticular arrays) have found many applications in imaging and nonimaging optics. A pair of lenslet arrays permits efficient uniform illumination for display systems and lithography [1], and recently microlens arrays have found application in condensing sunlight for photovoltaic conversion [2]. One of the notable early citations of integral image synthesis was by Gabriel Lippmann, using a lenslet array to construct a stereoscopic image with multiple subimages [3]. The plenoptic camera has leveraged the ray mapping of lenslet arrays for collecting three-dimensional information for post-imaging focus adjustment. Ng and colleagues go beyond computational refocusing to correct aberrations in an adjoining optical system with light field plenoptic photography [4].

Massof and colleagues show a spherically tiled system of Fresnel lenses centered with respect to the eyeball [5]. The spherical symmetry of the tiled lenses is cited by Massof to increase the field of view while maintaining resolution and limiting distortion. In later work, Lanman and Luebke of Nvidia have demonstrated application of lenslet arrays for highly compact VR HMDs [6]. We claim that a combination Lanman and Massof’s work, i.e. decreasing the size of the tiled lenses into lenslets tiled onto a curved surface, can not only further reduce the lenslet aberrations but also largely increase the field of view of a lightfield VR system. Assuming current progress in display technologies continues to decrease pixel size and increases display flexibility, curved displays with small pixels and appropriately compensated back-illumination will become increasingly available. Similarly, advances in manufacturability of micro-optics will enable increased realization of richly parametrized optics such as specialized lenslet arrays. This leads the current study to analyze monolithic spherically curved lenslet arrays as shown in Figure 1. The back surface has the display coupled into the lenslet array, and the concentric front surface has the lenslet array. Inherent symmetry of the system allows for an arbitrarily large field of view with improved image quality and illumination. This is compared with the planar array in the design methods.

Figure 1.

Schematic rendering of a curved monolithic lenslet array. Light is emitted from the back spherical display surface (hidden) through the front spherical lenslet array. Both dominant surfaces are concentric with the center of the eye. Design of the system is chosen in the cartesian YZ plane.

2.

SYSTEM LAYOUT

This section discusses the layout, assumptions, and basic parameters of the spherical lightfield VR system. From the direction of the eye, the system is comprised of a lenslet array on a spherical surface and a spherical display. Lenslets in the array have spherical shape and the array and display are concentric about the rotational center of the eyeball. Lenslets are arranged in azimuth/altitude tiling along the spherical surface, in a pseudo-square fashion corresponding to the rectangular tiling of pixelated displays. The y-axis is defined as the reference pole for altitude and azimuth coordinates. Because a 2D cartesian display mapping onto spherical azimuth and altitude does not preserve equal spacing across a wide field, an actually manufactured system would have to consider tiling requirements more carefully. Tiling is a practical consideration specific to manufacturing details, so the design study commences assuming representation in the cartesian YZ plane is sufficient. Similar to Lanman and Leubke, the design has a virtual image one meter away from the lenslet array, which implicitly constrains the distance between the display and the lenslet array given a lenslet focal length.

Distance between the lenslet array and the display (D_O), eye relief of the system (D_ER), and radius of the eye (R_eye) are defined on a YZ section of the system as shown in Figure 2. Since the lenslet array and the display are concentric about the center of the eye rather than the pupil, a user of the system cannot see the full rendered field of view at once. The design essentially creates a floating window within the rendered field of view, and the window appears to move with the rotation of the eyeball, always centered on the gazing direction of the eye. The floating window samples different parts of the full rendered field, as shown in Figure 3. With small lenslets and spherical symmetry, the highest resolution is always at the center of the gaze of the eye. This is discussed in the design methods.

Figure 2.

YZ plane view of the spherically symmetric lightfield VR system. The lenslet array on the spherical surface is placed in front of the eye, and a spherically symmetric display is behind the lenslet array. The system is concentric on the center of the eyeball, so the rays far from the lenslet axis (red) will be blocked by the pupil and the rays close to the axis (green) contribute to the image, leaving a circular floating window of view. These contents far off axis are seen by rotating the eyeball.

Figure 3.

Visualization of the floating windows for both eyes. The red dotted circles show the size of the floating window field of view shared by both eyes. The grey background is the binocular rendered field of view, containing green and blue object points. At left above, when the eye is focusing on the blue dot, the green dot is barely resolved because it appears on the edge of the floating window. Likewise, at right the gaze centers on the green dot and the blue dot is barely resolved.

3.

DESIGN METHOD

The design strategy is rooted in simultaneous improvement of two existing approaches. Lanman and Leubke with their planar lenslet array lightfield display and Massof with sphere-tiled lenses both present improvements with their solutions. Both approaches to HMD design have merits, and the proposed design method leverages all the merits. Understanding the spherical lenslet HMD design is the topic of the first section of the design method, and the second section outlines how the method proposed is an improvement over the existing approaches.

3.1

Specifications of the system

In the design process of the spherical lightfield VR system, the angular/spatial resolution and the size of eyebox are closely related to the size and effective focal length of lenslets. In this section, a set of equations defines the relation between these parameters and explore the limitations.

The lenslet array design ensures that each lenslet corresponds to an independent fraction of display (subobject) behind it. In the concentric shell design, the lenslet and its corresponding subobject share the same solid angle assuming a perfect tiling condition, i.e. the subobjects have no overlap or gap between them on the display. The size of subobject corresponding to each lenslet is given by

where w_O is the full size of a perfectly tiled subobject and w₁ is the full width of a single lenslet. These parameters are shown in Figure 4 with half angular size of eyebox θ and projected angle of the subobject α. D_O is the object distance, D_ER is eye relief and R_eye is the radius of the eyeball as above. Consequently, the subobject pixel count (N_p), a metric of spatial resolution, is

where p is the pixel size. Notice that for a uniform lenslet array on a spherical surface, the subobject size remains the same for the center of gaze across the whole field of view.

Figure 4.

Designation of system parameters. Subobject of full width w_o has half angle α when projected to the eyeball through a lenslet of width w_l. The range over which the eye can move and still register an image through a particular lenslet is the eyebox. This eyebox is specified to first order by w_eyebox. The half angle of the floating window θ is the angular representation used to represent the eyebox.

One of the most important specifications for HMDs is the angular resolution. In lightfield displays, the pixels are magnified because the display is placed close to the focal plane of the lenslet array. The pixel angular resolution on the subobiect is given by

The half angular size of a subobject is given by

The eyebox of the system is defined as the range the eye can rotate while still seeing the central subobject. To first order, the projected side length of a subobject on the eye is an eyebox metric

Since the system is concentric about the center of the eye, the eyebox has the same size for all subobjects. The calculation of projected side length assumes the eyebox is a plane mapped onto an eyeball, which is a good metric for quick analysis but is incompatible with the sphere geometry. A more applicable angular metric for evaluating the eyebox is the angle that eyeball can rotate while still seeing a specific subobject. The half viewing range of the eyeball to a specific subobject gives

where θ is the half angular size of the floating window.

4.

TRADEOFFS AND SIMULATIONS

The discussion in this section assumes an eye relief of 25mm, eye radius of 12 mm, and pixel size of 5 μm. The field of view can be extended arbitrarily to the limit of manufacturing constraints. The plots in Figure 5 reveal tradeoffs to be considered during the design process.

Figure 5.

System tradeoff plots. System parameters: 25 mm eye relief, 12 mm eye radius, 5 μm pixel size. Plots display pixel angular resolution (a), subobject pixel count (b), form factor (c) projected eyebox (d), angular size of floating window (e), and diffraction limited spot size (f), as functions of focal length and lenslet size (diameter). Plots are cut off where not defined or limited by fundamental factors. Diffraction limit is constrained less than one pixel per diffraction spot diameter at 587 nm. The eyebox cannot exceed the size of the eye. Balancing form factor with angular resolution constrains the lenslet focal length, and balancing eyebox with diffraction limit constrains the speed of the system. The optimal speed is between f/2 and f/4.

Figure 5(a) shows the pixel angular resolution is only a function of lenslet focal length, and the longer focal length provides better angular resolution. Figure 5(b) shows that for the perfect tiling condition, lenslets with larger sizes have higher pixel density and better spatial resolution in each subobject. Figure 5(c) shows form factor is driven directly by the focal length of the lenslet array. Figure 5(d) and (e) show that both projected side length of eyebox and the angular size of floating window have quasi-linear relation with the f/number of the system, and faster system has larger eyebox. The upper limit of eyebox size is the size of the eyeball. The f/number is limited by diffraction as shown in Figure 5(f) where single pixel per diffraction spot diameter is assumed.

The tradeoff plots show spherical lenslet array systems with long focal length and large lenslet size can provide better angular and spatial resolution with larger eyebox. Taking this argument into extreme, however, turns the system into a simple rotational symmetric magnifier system with large form factor and a significant amount of field aberrations. Focal length of the system is restricted by thickness of the system and the amount of field aberration in the system. The size and speed of lenslets are constrained by eyebox size and the diffraction limit. Lenslets faster than f/2 do not improve the eyebox, and slower than f/4 the diffraction limited spot is too large. In current configuration, an optimal design has focal length at about 6 mm and lenslet size about 3 mm.

An image simulation of the spherical light field system is performed to show the field of view experienced by the user. Tradeoff analysis shows a lenslet array bounded between f/2 and f/4 has a limited floating window size, so an 80 degree floating window is simulated on a 110 degree static field of view. Figure 6(a) shows the source picture of a camera man to be used in the simulation. The image is separated into overlapping subsections of the field and tiled on a spherical display without gap. Figure 6(b) shows a projection of the tiled subobjects on a spherical display. The simulated field of view of the system is shown in Figure 6(c). The central vision, corresponding to contents within the floating window, suffers minor resolution loss from pixel size limitation. The subobjects outside of the floating viewing window remain blurry and highly vignetted until the user looks in that direction, shifting the floating window onto that section of the field.

Figure 6.

Image simulation and tiling of subobjects. Overlapping subsections of the source picture in (a) is tiled on a spherical surface in (b), and an 80 degree floating window on a 110 degree static field of view is simulated in (c). The clear region in the simulated field of view is composed of the subobjects in the floating viewing window. Annular blurred region in the simulation is designed to match the lower resolution periphery of the eye.

5.

CONCLUSION

Near-eye light field HMD using planar lenslet array is considered one of the most promising technologies in virtual reality industry. Comparing to current designs using simple magnifiers, the lenslet array can obtain a compact form factor while efficiently expanding the field of view. However, the planar lenslet array geometry is incompatible with the rotationally symmetric geometry of the eye and limits further field of view expansion. The present design demonstrates a spherical lenslet array with a spherical display in concentric configuration can obtain a wide field of view, high spatial resolution, large eyebox and relatively compact form factor. The geometry of the system ensures high resolution at the center of the field of view wherever the eye tracks.