1.

INTRODUCTION

While mixed reality lightfield head-mounted displays with eyeglass-like appearance and full 154° per eye field-of-view (FoV) is considered as the ultimate specifications of future head mounted displays, in practice, the advancement toward such form factor and capabilities has been delayed by numerous engineering challenges. In the last decade, transparent slab waveguides have paved the way for smaller formfactors [1], and active optical components and computational methods have enabled low-resolution lightfield capabilities [2]. While there have been many studies that have proposed innovative solutions to these challenges, there have been very few studies on what are the actual fundamental trade-offs and limitations enforced by physics of light, geometry of human eye, and human visual perception limits.

This study discusses and compares the fundamental limitations and trade-offs that exist between different parameters of an arbitrary augmented reality display based on passive optical elements and pinpoints several intrinsic limitations and presents guidelines for designing optimal augmented reality experiences. In summary we pin point seven rules that should be considered when designing an augmented reality headset with passive optics and we present a theoretical framework for designing ultimate light-field experiences based on sparse monocular depth allocation.

2.

FUNDAMENTAL LIMITATIONS SET BY PASSIVE OPTICS

Both waveguides and visors are passive optical elements that convey or form the image that is generated by an external electronic source (usually either a projector, a micro display, or an LCoS display). This has been and most likely will be the dominant approach toward designing augmented reality displays in the coming decade for two major reasons: first, the advancements in lithography techniques and electronics have provided lower cost, higher resolution, and yet smaller form factors for such image engines, which makes them appealing for wearable headsets; and second, transparent electronics and other active optical components (e.g., tunable lenses, switchable Bragg gratings, etc.) are still not fully mass-producible for display purposes and they typically suffer from diffraction, speed, or efficiency issues.

Although appealing for mass-production, using an external electronic image engine with a set of passive optics to deliver the image to the eye comes with several fundamental drawbacks:

2.1

Rule 1: Sub-unity MTF effect

The image quality delivered through passive optics to the eye cannot be better than the original quality of the image produced by the electronic engine. This is a fundamental degradation that happens naturally due to absorption, spectral characteristics of components, inaccuracy in surfaces, unwanted reflections, and many other deficiencies in passive optical components. This is the same reason why a camera lens can never capture an image that is better than the original scene in terms of spatial or color accuracy. Therefore, based on this principle, it is impossible to get better pixel accuracy, color accuracy, or brightness than the original image that was produced by engine using passive optics such as waveguides or visors. In practice, the image quality can only get worse, as there are no perfect modulation transfer function (MTF) optical components (Fig. 1 (a) top).

Figure 1.

(a) Top diagram: the image accuracy will never be better than the original image source. Bottom diagram: as the number of perturbation phenomena increases, on average the color accuracy drops. (b) Due to reciprocity, a more efficient optics appears darker from the outside world. (c) Schematics of light exiting an arbitrary passive optics of an AR headset. d is the exit pupil extent, l is the minimum eye relief, θ is the FoV.

2.7

Rule 7: Additive augmentation reduces image dynamic range or contrast

The additive nature of augmented reality with passive optics means that the light intensity from visual content is added to the light intensity from outside environment. This reduces the dynamic range of the image (typically by orders of magnitude) as below:

In equation (1) D_total is the total dynamic range of the image seen through the glasses which includes both the scene and the augmented image. Here I_AR and I_SC is the augmented image and environment image intensity as function of x and y and the Max and Min subscripts indicate the maximum and minimum of these intensity values. The image addition or transparency coefficient 0<α<1 depends on the transparency of the glasses; in and ideal hypothetical case α = 1 and the glass is fully transparent with no surface reflection. In practice this coefficient is between 0.3 and 0.7. Equation (2) indicates the augmented image dynamic range D_AR. This value is more important than D_total since due to uniform attenuation of the environment intensity α with the glasses the dynamic range of the environment image is not impacted by the passive optics. Here I_ave is the average intensity of the environment. For example, this average increases significantly at outdoor compared to indoors. As noted, the D_AR is reduced notably for a more transparent glass. At completely occluding case where the glasses are dark (α = 0) the dynamic range of augmented image reaches the dynamic range of the electronic engine (D_El). So if one starts with an electronic engine with very high dynamic range, the additive nature of passive optics is still expected to reduce that dynamic range significantly. Furthermore, there is a direct relation between dynamic range of augmented image D_AR, maximum optical power of the engine P_ARMax (engine refers to the light source that feeds the passive optics), the efficiency of the optics η (such as visor or waveguide) to bring engines light to the eye, the transparency coefficient α, environment average brightness I_ave, and area of the eye box A_box as indicated in equation below.

Equation 3 shows that dynamic range has a reverse and, in most cases, nonlinear relation with transparency and eye box area. In other words, based on Eq. (3) with the same peak power one can expect a lower dynamic range for a more transparent looking AR glasses or glasses with larger eye box.

As noted in this section, the laws of physics impose a set of fundamental limitations and trade-offs on performance and form factor parameters of AR displays. These limitations are independent of the optical design or fabrication technology used to make the passive optical surfaces. We need to note such guidelines when passive optics is considered for engineering the augmented reality headsets. However, in addition to such guidelines there are some guidelines that are imposed by human visual perception.

3.

ULTIMATE SPECIFICATIONS SET BY HUMAN VISUAL PERCEPTION

Human visual perception is rather an extended topic with rich literature focusing on different parameters [3-10]. In most cases its extremely difficult to pinpoint one fixed number that models a certain aspect of human visual perception. For example its difficult to assign one (x,y) resolution number to the eye since eye is really not a fixed camera and the retina sensing mechanism is vastly different from that of an electronic sensor. To make things even more complicated the perception of the image is rather subjective as the signals are processed with brains and both eye and brain processing is known to vary slightly in between population and even with age. However, such complexity does not mean that there is no way to estimate or characterize the visual perception over a defined population. We like to approach visual perception from display perspective rather than phsychophysics perspective. For instance, we would like to know what is the (x,y,z) resolution that would saturate the average eye at age of 30; what is the field of view that would be sufficient for most population, etc. Specifying each of these parameters with indication of their dynamics in one study is not possible so we would use the estimates that has been found in different studies on spatial, temporal, spectral and aberration modeling of human eye [5-10] and just note the numbers briefly here. Table. 1 shows the estimates that would fully saturate the human visual perception for an average healthy eye.

Table 1.

Parameters for ultimate augmented and virtual reality experience based on human visual perception and existing standards in display industry. There is not much information on the depth.

As noted in Table 1. while there are at least rough estimates for majority of parameters there is less attention being paid to monocular depth perception specifically depth perception from display perspective. Since majority of light field display modalities have been rather recent [11-15]; there is no clear notion of what is the limit of human eye saturation in depth and what are even elementary lightfield depth levels that should be created in a discrete quantized level to provide the richest experience.

Monocular depth perception has been pondered over in psychophysics, human perception, neuroscience literature with completely different approach [16-19]. Although informative, these psychophysical studies are human centric with no specific compatibility or interest in digital augmentation of human perception with emerging display technologies. For example, while there are scattered studies on human eye depth of field for different age groups and pupil conditions, there is no general or scientifically grounded conception of a technical guideline or tool to design a VR or AR displays that would satisfy such perception acuity.

In order to quantize the monocular depth levels one has to consider eye diopter range, depth of field and its relation to pupil size. Each of these parameters varies with lighting condition and age and can dramatically change the number of depth levels that are distinguishable. If the largest diopter range reported in the literature for very young eye (15 diopter [20]) is divided by the shallowest depth of field reported in the literature (0.15D full width half maximum [21]) then the 100 depth levels are the absolute maximum number that human eye at age of 10 can distinguish with 6-8mm pupil size. However, if one assumes an average of 6D range for adults with 0.15D depth of field, this maximum number of depth levels reduces rapidly to 40 levels which is a more practical estimation for average young eye. The depth of field significantly varies with pupil size [21] and the accommodation range significantly varies with age [23]; therefore, its essential to consider these two parameters in laying out the physical localizations of these depth levels. Here we use an iterative method to localize the depth levels up to 10 meters.

DOF (Depth-of-field) for 2mm, 4mm and 6mm pupil diameter has been previously studied in [21,23]. Anderson et al. [24] has used the objective method to measure the accommodative amplitude in a wide age range of individuals, and has given a sigmoidal function fit to the measured data. The function was used to find the max accommodative amplitude. We started at the nearest focal plane given by this max accommodation and iteratively find the next focal plane at a step of DOF from [24]. The iteration stops when the focal plane distance is larger than 10m (Fig. 2.)

Figure 2,

Monocular distinguishable focal planes with age and pupil diameter variations. Focal plane distances are within 10 meters. Color bar shows the corresponding total eye diopter (60 from a relaxed eye+accommodation). As one grows older, the distance to the nearest focal plane becomes larger and number of total focal planes decreases, e.g., with pupil size equal to 2mm, one is able to distinguish 13 focal planes at age of 10 but can distinguish only 2 focal planes at 50.

As noted in Fig. 2 the ultimate number of depth levels that would saturate the monocular perception varies from 40 at darkest environment (pupil diameter of 6mm) down to only 2 or 3 depth levels at bright environment for elderly eye. To find an estimate of number of levels needed in average population in average condition we considered the average display brightness of 250nits that would contract the pupil to 3mm [23] and calculated the depth locations based on age as in Fig. 3. As noted for average age of 30-40, which includes the median age of majority of countries, the maximum number of depth levels that is distinguishable is only 10-12 levels. This number can be substituted for the question mark on Table 1.

Given the statistics of eye diopter across different ages and based on daily task operations one can use optimization to find the priority of allocating depth levels. This is significant in design of 3D displays as the bandwidth is limited and only limited number of monocular depth levels can be rendered. We show this optimized localization of the depth levels in our future studies.

Figure 3,

Monocular distinguishable focal planes with age at 3mm pupil diameter. Focal plane distances are within 10 meters. Color bar shows the corresponding total eye diopter (60 from a relaxed eye+accommodation).

In conclusion, through this brief study we have highlighted seven fundamental limitations that is imposed by physics of passive optics and highlighted the ultimate specifications that is needed based on the saturation of human visual perception. We found that only about 10 depth levels are enough to fully saturate the human monocular depth perception at average display brightness.