Daylight luminance requirements for full-color, see-through, helmet-mounted display systems

Abstract. When color is implemented in helmet-mounted displays (HMDs) that are eyes-out, see-through displays, visual perception issues become an increased concern. A major confound with HMDs is their inherent see-through (transparent) property. The result is color in the displayed image that combines with color from the outside (or in-cockpit) world, producing an image with additive color. As luminance of the HMD imagery is reduced, the color separation between the HMD imagery and the background is also reduced. It is because of this additive effect that luminance contrast is so vitally important in developing HMD standards for color symbology. As a result, this paper identifies luminance requirements for full-color HMDs based upon two lines of investigation. The first is based on a study of white symbology against natural static backgrounds, where the quality of symbology was judged to be a function of not only the background luminance but also of the background complexity as well. The second is based on an evaluation of the complexity inherent in natural backgrounds and from this investigation, a predictive curve was found that describes the complexity of natural backgrounds as a function of ambient luminance.


Introduction
For symbology to be viewed in a see-through helmetmounted display (HMD) or head-up display (HUD), the luminance of the symbology must be sufficient to distinguish it from the see-through background. This is true whether or not symbology is displayed on a monochromatic or full-color display. When the contrast of the transparent symbology is sufficiently high, the symbology appears as an overlay on the ambient scene. In order for an HMD or HUD to be usable in an operational environment, the luminance requirements must take into consideration the type of displayed imagery (e.g., symbology, situational maps, target sights), the tasks (e.g., targeting, navigation, obstacle avoidance), the operational setting (e.g., day/night, terrain features), additional hardware (e.g., visors, windscreens, laser protection), and other considerations. For a color HMD, symbology color overlaid on an ambient scene should consider luminance and color contrast, which requires information about the spectral content of the landscape or ambient scene. However, optical designers must consider all terrain features when designing a see-through optic as the military can be deployed to any geographical location around the globe; this fact may simplify the development of luminance requirements for see-through, fullcolor HMDs. The reason that this will simplify the development of a contrast requirement is that luminance contrast, and not color contrast, is the determining factor.

Modeling Helmet-Mounted Display Symbology
As an initial step in defining the luminance requirements for daylight symbology, Harding et al. 1 processed simulated images of white symbology (simulating a full-color HMD with all three primary colors displayed) overlaid on selected static natural backgrounds (Fig. 1). Observers evaluated the quality of symbology (Table 1) in "contrast correct" images of symbology overlaid over eight natural backgrounds (an example is shown in Fig. 2), one artificial background (artificial clutter in Fig. 1), and one uniform field. The overlaid symbology shown in Fig. 2 was created by use of a software model 1 that applied appropriate luminance scaling to yield a symbology image that has been added to the background image. In this experiment, the complexity of the background image was of paramount importance when determining the minimum luminance requirements for see-through symbology. This finding is in general agreement with other studies [2][3][4] of transparent text against complex backgrounds. The luminance of the background image was of less importance.
For each background scene, 20 symbology images of varying contrast were evaluated by observers. Following a training session, observers judged the quality of symbology for each of the 200 images in three test sessions. All observers were required to pass an intermediate-field acuity test (distance of 22 in.) and a Farnsworth 15-Hue color vision test. There were no gender or age-specific requirements. Twenty volunteer observers were selected for the study, however, for an observer's data to be included in the final data analyses, critical judgment criteria were established and only data from seven of the observers were selected for final data inclusion (see original paper 1 for selection metrics). For each of the 200 images, an average Michaelson contrast ½ðL max − L min Þ∕ðL max þ L min Þ was calculated for each image. To make calculations easier, block symbology images were processed in exactly the same fashion as the symbology image. For example, for each of the 20 blocks shown in Fig. 2(b), L max equaled the average luminance for each of the 20 symbology blocks. Likewise, L min was the average calculated luminance of 20 areas, of the same size adjacent to them.
The size of each of the background images was 640 × 480 pixels. The patch size was 50 × 50 pixels. Luminance values were calculated for each subpixel using the following equations: E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 2 ; 3 2 6 ; 3 1 3 where R, G, and B were the red, green, and blue 8-bit subpixel values (i.e., 0 to 255). GAMMA was the computer monitor's Gamma, and in these calculations was set to a value of 2.2 [ Fig. 3(a)]. The coefficients 0.23, 0.67, and 0.10 provide an average fit to the relative luminance contributions of the subpixels. The luminance of each pixel, L pixel , equaled the sum of the subpixel luminances and E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 2 ; 3 2 6 ; 1 6 9 L patch ¼ ðΣL pixel Þ ÷ N; where L patch was the average luminance of the patch and N was the number of pixels in the patch (2500). For pixels with RGB values of 255, 255, 255, L PIX is equal to 1.0. The highest calculated pixel luminance in the background image was scaled to a peak simulated luminance which differed for each contrast condition. Although simulated imagery could not cover the photopic range expected in the cockpit  [Sony monitor used in the experiments had a peak luminance of 28 foot-Lamberts (fL)], overlaid symbology calculations produced contrast correct images. Given the hardware transmission spectra simulated in the model, 1 simulated luminances at the eye ranged from a low of 7 fL for the Ground Clutter 2 image to a high of 6198 fL for the uniform background. To evaluate visual sensitivity differences over the simulated photopic ranges and the actual display output, we used Barten's model 5,6 to calculate static contrast sensitivity for average luminances of 10, 100, 1000, and 10,000 fL [ Fig. 3(b)]. Symbology characters were 37.5 arcmin wide with a 7.5 arcmin gap. The gap size (if the gaps were continuous) corresponds to a spatial frequency of 4 cycles∕ deg. Calculating the contrast threshold range (1 ÷ contrast sensitivity) at 4 cycles∕ deg reveals an insignificant difference of <0.01% in the four curves suggesting that photopic simulations can be made at the monitor's reduced luminance. The minimum contrast for symbology was judged to be equal to an observer rating of 4 on the seven-point scale, as described in Table 1. Harding et al. 1 found that the following equation provided a reasonable fit for describing the luminance required for a minimum contrast rating of 4.0: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 3 2 6 ; 3 1 5 where L B is the luminance of the background and B SD is the standard deviation of the background luminance described as a percentage of the mean luminance. One of the backgrounds used in the study was a uniform field (standard deviation ¼ 0). For a uniform background, the minimum symbology luminance was found to be 10% of the background luminance, hence the 0.1 constant in Eq. (1). The 1.42 scalar in Eq. (1) was derived by averaging best fit scalars for each of the eight natural backgrounds and the artificial clutter image. When Michaelson contrast was plotted as a function of observer ratings, the curves for different background images were quite disparate (Fig. 5). When background complexity was added as a codeterminant, the plots became much tighter [see Fig. 7

Background Clutter in Natural Scenes
Although the natural background images were deemed representative, the imagery was static and not dynamic. During actual flight, flight symbology would appear overlaid over a  changing ambient scene. Motion tends to blur or reduce the local variations in an ambient scene (especially in nap of the earth flight), and the background relative to the symbology has a temporal modulation component. At minimal contrast, symbology characters would likely appear recognizable in one instance and perhaps partially disappear in the next instance as the complexity and luminance flow of the ambient image waxed and waned. There is certainly a dynamic feature to the visibility and quality of symbology, and taking motion into account would likely yield more accurate results than simply evaluating static symbology. Analyzing the effect of motion on symbology visibility and quality, however, is beyond the scope of this paper. Since the quality of symbology is based on the visibility of each character or indicator against its background, we expressed the B SD in Eq. (1) as the average percent standard deviation of small patches of background. In this way, the background complexity, as measured by the standard deviation, would always be relevant to the overlaid symbology. We coined the term "local contrast" to describe the contrast between the luminance within a character or indicator divided by the average luminance of adjacent patches that subtended an area equal to about the size of a symbology character. 5,7 By setting the maximum pixel luminance in each of the nine images shown in Fig. 1 to 5000 fL, a patch analysis distribution of luminance versus complexity produced scatter grams whose upper envelope could be well described by the following equation: (2) where lnðL B Þ is the natural logarithm of the background luminance in fL and complexity is in percent standard deviation. Below about 2200 fL, the limiting equation yields percent standard deviation measures that are greater than 100%. Above 6000 fL, the curve produces a negative complexity value. However, over a large photopic range of 1 to 6,000 fL, the approximation is reasonable. Of the data plotted in Fig. 4, over 99.9% of the patch complexity data falls on or below the curve from Eq. (2).

Observer Ratings of Symbology
In the original study, 1 observers evaluated the quality of symbology in 200 images, 20 images per background image. Average contrasts were calculated for each of the 200 images, and the contrasts varied widely depending on the background images. The average Michaelson contrast was calculated for each of the images and these contrasts were plotted as a function of the average observer ratings. Figure 5 shows the results for all 10 of the background images. Least square power functions were fit to each image set (R 2 > 95% based on curves fit to data points at or below an observer rating of 6.5). For each background image, observer ratings of 4, 5, and 6 were calculated from the curve fits. For six of the images, observer ratings of 4 or lower were not observed, and extending the curve fits allowed calculations of the minimum contrast rating (i.e., see data for images "cloud 1," "cloud 2," "horizon 2," "horizon 3," "horizon 4," and "uniform" in Fig. 5). To assess the relationship between the contrast required for an observer rating of 4, 5, and 6, observer data for each of the images were  Contrast requirements as a function of B SD for observer ratings of 4, 5, and 6. Each set of data has 10 data points representing the 10 background images. 1 The straight lines are linear fits to each of the observer ratings.
averaged and plotted. Figure 6 shows the contrast requirements for each of the 10 images as a function of B SD . For purposes of this discussion, we will term observer ratings of 4, 5, and 6 as minimum contrast, average contrast, and good contrast, respectively. Linear curves of the same slope were fit to each of the ratings in  (5) can be calculated over the range of 1 to 6000 fL by simple substitution. Before we can make these calculations, however, a cap must be placed on the B SD values since at low luminance, the envelope curve is characterized by increasingly high B SD values. These values represent contrast variation in the shadows of most natural images and because of this are likely not readily apparent to observers due to a higher light adaptation level of the visual system. Even the darkest background image, "ground clutter 2," has an average calculated luminance value of 877 fL based on a 5000-fL peak luminance. At low luminance values, the envelope curve would require contrasts higher than 1.0, which by definition are not achievable. To overcome this difficulty, we capped maximum contrast at 0.8 and this was based on observer ratings. Evaluating Fig. 5 as a basis and noting the good contrast data points, all average contrast values were below 0.8. Using 0.8 as a contrast limit places an upper limit on B SD to about 1.28, 1.07, and 0.76 for minimum, average, and good contrast, respectively. Using this contrast limit, Fig. 7 shows the contrast requirements for the three contrast conditions as a function of L B . To calculate the luminance requirement for symbology, (L sym or L max − L min ) based on the data, shown in Fig. 7, requires assumptions about the luminance distribution within L B . Within a background scene, if we assume that pixel luminance values are normally distributed, then the middle luminance (L mid ) can be calculated as follows: where C is the contrast constant in the minimum, average, and good contrast linear equations above. Using this contrast cap, Fig. 8 shows the luminance requirements for the three contrast conditions.
The three contrast curves merge to a straight line at low luminance values and this is due to the limit placed on B SD , as seen in Fig. 7. We termed this straight line the 0.8 Michaelson contrast line. Each contrast curve peaks at an intermediate luminance level due to the fall-off in B SD with increasing luminance. Of note, the peak luminance values did not change even when the Michaelson contrast was capped at a higher 0.9; thus, we feel confident that this method yields luminance requirements that are trustworthy.
In the original study, 1 contrast ratios were calculated from the block symbology images and Eq. (1) provided a reasonable fit to that data for an observer rating of 4.0. Using that equation in association with Eq. (2) yields a peak L sym of 2278 fL. This luminance estimate is about 20% lower than the 2840 fL found using the Michaelson contrast curve fit. The method in which the 1.42 scalar was derived is likely not as accurate as it represents an average of exponential curves fit to the contrast ratios for each background image.
The peak luminance values in Fig. 8 were based on linear curves fit to the data in Fig. 6. When calculating maximum luminance requirements for symbology, it would be better to use curves fit to the maximum contrast data in Fig. 6 rather than using curves fit to the average values. This can be achieved easily by simply scaling the three linear curves upward until all data points are either on or below the straight line fit. Scaling is performed by simply increasing the value of the constant in Eqs. (3) to (5). Table 2 shows the results of this process along with the peak L sym luminance requirements for each contrast condition. For the good contrast condition, L sym peaks at 5470 fL, and this value will form the basis for our discussion about color.

Color Symbology
In summary, the 5470-fL peak luminance value from Table 2 was calculated for white symbology against eight natural backgrounds, one artificial background, and one uniform field. This does not take into account transmission characteristics of military hardware even though military hardware was modeled during the collection of the original data. The images produced by the model were evaluated by observers, and calculations were only made on these standalone images with contrast-correct representations of symbology overlaid over each background. Even though a full-color HMD was modeled, only white symbology was evaluated as it represented the highest achievable luminance contrast.
When developing see-through HMD luminance and color contrast requirements, consideration of natural terrain features becomes an obvious concern as background luminances and color add to the luminance and color of the symbology. In choosing symbology colors, each must have sufficient contrast against any background color to be easily distinguishable and recognizable. When considering sufficient contrast of color symbology against natural backgrounds, it may be beneficial to use the luminance requirements for white symbology as a guide. This would allow us to calculate the HMD luminances for various colors that might be used in a see-through HMD. For example, Fig. 9 shows the chromaticity coordinates for a full-color liquid crystal display (LCD) using the 1976 Comission Internationale de l'Eclairage (CIE) chromaticity diagram. For this particular display, the red and green chromaticity coordinates are near the monochromatic border as the spectra have fairly narrow bands at 612 and 544 nm, respectively. This LCD provides a good example of the relative luminances expected in a full-color HMD. Table 3 shows the primary values for each of the colors plotted in Figs. 9 and 10. The relative luminances of the three primaries (red, green, and blue) were chosen by industry such that their summation (with each primary set to the same value) would yield the achromatic grayscale or shades of gray from black (0,0,0) to white (255,255,255). The white value is represented by the middle data point in Fig. 9. Table 4 shows calculated luminances based on the relative luminances of the LCD example. By setting each color equal  The blue display spectra mostly stimulate the short-wavelength cone whose population in the retina is sparse in comparison to the medium-and long-wavelength cones, and are thus not as suitable for distinguishing small characters. See-through HMD imagery is additive in that each displayed pixel is a function of the display as well as the ambient scene. As the ambient scene gets brighter or display luminance is reduced, the color of the displayed pixel increasingly takes on the color of the background. Because of this, it is difficult to discount the importance of luminance contrast above all other considerations. In other words, if we define color contrast as a separation between coordinates on a chromaticity diagram, then as luminance contrast is reduced, the color contrast is also reduced as the distance between chromaticity coordinates shrinks. 6 This concept is depicted in Fig. 11.

Hardware Considerations
Acquisition specifications for see-through HMDs should take into account the hardware that will be used with the HMD. Compatibility issues aside, windscreens, visors, and the HMD combiner lens will block some of the ambient light reaching the eye, and this light reduction should reduce the overall high luminance requirement. 8 For example, helicopter windscreens and tinted visors generally have a light transmission curve that is generally flat across the visible spectrum. If a tinted visor is likely to be worn during daylight hours, then the transmissivity of the windscreens and visors should be taken into account. In addition, HMD combiner Table 4 Calculated required see-through HMD display luminances (fL) for the seven colors shown in Fig. 10. Values were calculated relative to each individual color set equal to the peak 5470 fL luminance value derived above.  lens coatings provide some attenuation of ambient light transmission. For flat transmission spectra, assuming a 75% transmission of an aircraft windscreen, a 15% transmission of a tinted visor, and a 40% transmission of the HMD would yield a total light transmission of 4.5%. Applying this transmission to the 5470 fL value calculated above results in a luminance requirement of 246 fL for symbology. If red symbology was needed in a full-color system, the total output of the HMD would equal 966 fL, calculated from Table 4 above. HMD transmission spectra are normally notched at one or more HMD emission spectra peaks (e.g., at 543 nm for the P-43 phosphor used in the U.S. Army's AH-64 Apache HMD). This is done to improve contrast of the emissions against the ambient scene. Certainly, notch filters reduce the overall transmissivity of the combiner lens, which will improve contrast. However, under certain circumstances, it has little effect. For instance, the example shown in Fig. 11 would not be improved by a typical narrow-band notch filter as the emission spectra is green and the background is composed of red and blue light.

Discussion
The contrast requirements provided here were based on observer ratings of the quality of symbology in static imagery. 1 The imagery consisting of eight natural scenes, a uniform field, and a background image composed of artificial clutter was superimposed with symbology over a rather large range of contrast conditions. Ratings of the quality of the symbology were used to generate equations for minimum, average, and good contrast as a function of luminance and background clutter. These equations were then applied to an envelope curve that describes possible background clutter as a function of daylight luminance. 5,7 The results of this application produced daylight luminance requirements for minimum, average, and good contrast. Using the good contrast value of 5470 fL produced the results in Table 4, which describes the luminance requirements for symbology of different colors (symbology consisting of the three primary colors and their combinations).
The results in Table 4 were based on the emission spectra of a typical LCD. As manufacturers will normally adjust the contributions of the three primaries, at their maxima, to produce white light of a given color temperature, similarly scaled contributions from the three primaries will likely be used regardless of the light source.
The question that likely arises from these results is how robust is the luminance requirement determined in this paper. The imagery that observers evaluated was well-controlled, with a calibrated monitor and luminances calculated based on photometric assessments. The imagery was static and dynamic imagery may somewhat reduce the luminance requirement but will not likely increase it. What may increase the luminance requirement is displaying imagery other than symbology (e.g., sensor imagery, synthetic imagery, tactical maps). Aviators are very familiar with the symbology set used in their aircraft. They know the function and position of each element, and this knowledge assists them in reading the symbology when contrast conditions may be poor. On the other hand, other imagery has an unknown quality about it and to correctly decipher information content requires good contrast conditions. In a preliminary investigation, Harding et al. 9 found that, indeed, map imagery required greater contrast to interpret, especially against a cloud image (high luminance, low clutter), where map imagery needed about twice as much luminance contrast compared to symbology. Against high-cluttered backgrounds, symbology and maps equally required high contrast. Of course, this is a rather coarse estimate based on preliminary findings. We do, however, feel fairly confident that for HMD symbology, the data provided here should assist in establishing general guidelines for HMD luminance performance requirements.

B SD Envelope Curve and Luminance
Distributions within Natural Scenes Equation (2) represents peak B SD as a function of luminances in natural scenes. We used this curve to derive the daylight luminance requirements for see-through displays. The equation provided a limit or envelope curve where 99% of the data plotted in Fig. 4 fell on or below the curve. The data from the artificial clutter image were largely responsible for setting the boundaries of Eq. (2). Based on a limited sample size of nine images, should we trust the curve to clearly reveal the limits of B SD over the expected daylight luminance range? To answer the question of background clutter, we chose two natural images that were composed of extremely high contrasts and high spatial frequencies. The images are shown in Fig. 12. The complexity of the image, B SD , is plotted as a function of scaled luminance. As stated earlier, Eq.
(2) provides a reasonable envelope containing over 99% of the complexity values from the two images. Equation (2) also assumes that 6000 fL is a reasonable peak luminance to use when determining operational light output requirements of HMDs. This value is an upper range compromise of the many background luminances encountered across fixed-and rotary-wing military flight environments. HMD imagery must be viewable against such varied backgrounds as grass, dirt, sand, water, snow, clouds, and sky. 10 While values in excess of 30,000 fL can be encountered (i.e., white cumulus clouds), in the high-altitude, fixed-wing environment, a 10,000 fL maximum luminance value representing a clear sky is frequently used as a representative background luminance in HMD calculations; 11 for the rotary-wing environment, 6000 fL (representative of a bright overcast sky) is typically used.

Limitations of Using Static Imagery
The background images used to evaluate the quality of overlaid symbology (Fig. 1) were static, i.e., no relative motion between the symbology and the selected background. However, in real-world HMDs, symbology is viewed against pilotage imagery, either synthetic or real-world (i.e., provided by a sensor mounted on the aircraft or pilot's head). In such scenarios, the symbology is viewed against a moving background, resulting in varying degrees of relative motion between the symbology and the background scene. In the simplest of these scenarios, the symbology may be viewed against a uniform sky or water background, which, having no spatial content, produces no perceived relative motion, and can be considered equivalent to the static case. However, in most flight scenarios, the background is a complex spatial scene and relative motion is present between the symbology and background.
HMD symbology may result in greater relative symbology motion than HUD symbology. Because the symbology block the real-world background scene, they are perceived as being nearer than the background, even though they are collimated. The motion of the symbology is perceived as faster even though it moves at the same rate as the real world. 12 It has long been known that relative symbology/background motion in HUDs can degrade retinal image contrast of either the symbology or the background scene (depending on viewer fixation), although performance effects are complex, depending on viewer task, symbology characteristics (e.g., luminance, shape, and size) and location, and background characteristics (e.g., clutter, luminance, and contrast). 13,14

Limitations of Using White Symbology to Determine Daylight Luminance Requirements
Although luminance and contrast requirements for the legibility of white symbology have been and are a continuing topic of investigation, such requirements for color symbology in HMDs have not been pursued as vigorously. [15][16][17] Metrics defining color contrast (and hence luminance) are more complicated than those presented previously where the contrast refers only to differences in luminance. Color contrast metrics must include differences in chromaticity as well as luminance. And, it is not as straight forward to transform chromatic differences into just noticeable differences (JNDs) in a perceived color space. This is due to a number of reasons. First, color is perceptually a multidimensional variable. The chromatic aspect, or hue, is qualitative and twodimensional, consisting of a blue-yellow axis and a redgreen axis. Second, the dimensions of saturation and brightness, as well as other factors, such as the size and shape of a stimulus, affect the perceived color and color differences. 18 As an example, an orange band against a grey background will appear "brighter" than an identical blue or green band at the same saturation value; even though a grayscale conversion may demonstrate equal luminances. 19

Displaying Symbology versus Imagery in a See-Through Helmet Mounted Display
Defining the luminance requirements for a color display using Table 4, for example, presupposes that the display was developed to yield a white balance when the red, green, and blue primaries are all set to their maximum level (e.g., 255 in an 8 bit-per-color display). Displays that are white balanced reproduce imagery that best mimics realworld colors. If the function of the HMD is to only produce symbology, proper white balancing may not be necessary. In laser-based scanning HMDs, for example, the short wavelength laser could be driven by a much higher voltage to offset the low sensitivity of the visual system to lower wavelength light. In this instance, the red and green lasers need not require a higher driver voltage thus, the high luminances shown in Table 4 for white symbology could be significantly reduced. However, using commercial display systems (e.g., LCDs, OLEDs) as sources in an HMD, you will most likely encounter displays that are purposely set to achieve proper white balancing.

Disclaimer
The views, opinions, and/or findings contained in this paper are those of the authors and should not be construed as an official Department of the Army position, policy, or decision unless so designated by other official documentation.