Modeling a color-rendering operator for high dynamic range images using a cone-response function

Abstract. Tone-mapping operators are the typical algorithms designed to produce visibility and the overall impression of brightness, contrast, and color of high dynamic range (HDR) images on low dynamic range (LDR) display devices. Although several new tone-mapping operators have been proposed in recent years, the results of these operators have not matched those of the psychophysical experiments based on the human visual system. A color-rendering model that is a combination of tone-mapping and cone-response functions using an XYZ tristimulus color space is presented. In the proposed method, the tone-mapping operator produces visibility and the overall impression of brightness, contrast, and color in HDR images when mapped onto relatively LDR devices. The tone-mapping resultant image is obtained using chromatic and achromatic colors to avoid well-known color distortions shown in the conventional methods. The resulting image is then processed with a cone-response function wherein emphasis is placed on human visual perception (HVP). The proposed method covers the mismatch between the actual scene and the rendered image based on HVP. The experimental results show that the proposed method yields an improved color-rendering performance compared to conventional methods.


Introduction
High dynamic range (HDR) imaging is a photographic technology that assembles and saves photographs of a static scene taken at different exposures in a radiance map, similar to the process employed by the human eye.HDR imaging has been developed based on visual adaptation and is a technique that mimics the human eye.The human eye readily captures a range of light intensities in multiexposure time and has a dynamic range of 6 orders of magnitude.Conversely, low dynamic range (LDR) devices have a range of 2 to 3 orders of magnitude. 1,2This leads to a problem termed tone mapping or tone reproduction 3,4 when displaying HDR data on LDR display devices.To display the original scene using the HDR representation on display devices with LDR, it is necessary to summarize the intensity range of the original image using the single extreme ratio between the maximum and minimum intensities. 5This ratio is the dynamic range of the image.
The goal of the color-rendering process is to maximize the perceived similarity between the actual scene and the displayed image.Therefore, tone-reproduction algorithms attempt to scale HDR data in order that the resulting displayable image preserves the characteristics of the input data, such as brightness, visibility, contrast, and appearance by controlling the scale of the given image.To address this issue, adaptive scale-gain retinex was proposed by Kotera and Fujita. 6In this model, to maintain the color balance, the surround image generated from only the luminance component was used for the R, G, and B channels.They also proposed an automatic adapted scale-gain weight-setting method.However, the computation of weights consumed a significant amount of time for a large Gaussian kernel size because it required a histogram luminance single-scale retinex corresponding to multiple scales.Therefore, the integrated-surround retinex method, which used only the relative luminance Y component of the YIQ representation instead of the three RGB channels, was proposed by Wang et al. 7 The result was stable and demonstrated high saturation as the local contrast of luminance preserving the color balance was enhanced using a multiscaled process.However, both methods were based on a slow gradient of the integrated surround in the center/surround process, thereby introducing halo artifacts.Further, by employing a single Gaussian filter, the chromaticity of illumination could not be removed, thus, the enhanced saturation was unnatural when compared with the original image.A local adaptation retinal model (LARM) was proposed by Wang et al. 8 based on the retinal model.LARM adopted a bilateral filter to reduce the halo artifacts (or ringing effects) in the resulting image.The local contrast was enhanced without halo artifacts compared to the original image.However, LARM was based on a nonlinear bilateral filter and a single luminance image to compute the surround image.Therefore, the local contrast in the dark regions could not be enhanced in the resulting image, thus the detail in the resulting image was difficult to distinguish with the human visual system (HVS).
Kuang et al. 9 proposed a new image-appearance model based on the HVS called iCAM06.It was developed for HDR image rendering.Cone and rod-response predictions addressed the mismatch between the displayed image and the rendered image using a chromatic-adaptation transformation in iCAM06.However, the luminance factor reduced the luminance adaptation in the cone-response function.Furthermore, stimuli of higher luminance than that of the adopted white could cause the response to approach the maximum level and consequently reduced the colorfulness. 10his paper presents a color-rendering model comprised of a tone-mapping operator and the cone-response function.This method is primarily processed in the XYZ tristimulus space of the input image.This method can manage the aforementioned problems that exist with the conventional methods.In addition, the linearity in the proposed tone mapping method is an important task to consider with one-to-one correspondence between the displayed and rendered images.Although the proposed method does not have a linearity, the goal of the proposed tone mapping method is to have a near linearity.In addition, the cone response function, which is based on human visual perception, is to address the color shift and color leakage by modifying the surround luminance factor (F ¼ 1 for average surround, and F ¼ 0.8 for dim or dark surround) in Ref. 1 and iCAM06, respectively.By using the modified luminance surround factor, a wide variety of surround conditions is automatically addressed to achieve excellent results for color constancy.In Ref. 1, the global tone mapping operator is just used to deal with the tone mapping problem and heavy time cost.Therefore, the method cannot address the local detailed variance information in the given image.The local standard deviation is used to overcome this problem.These issues are the noticeably main differences compared to Ref. 1 and iCAM06.Hence, the proposed method is capable of managing the problems such as color shift, color leakage, and color cast.
The tone-mapping operator produces visibility and an overall impression of brightness, contrast, and color in the given HDR image when mapped onto relatively LDR displays or printers.The tone-mapping image is obtained using chromatic (XYZ tristimulus value) and achromatic (absolute luminance Y component) colors.The adaptation can be considered equivalent to the dynamic mechanism of the HVS, which optimizes the visual response for a particular viewing condition.Dark and light adaptations refer to changes in visual sensitivity when the level of illumination is decreased and increased, respectively.Chromatic adaptation preserves the approximate appearance of an object.It can be described as an independent sensitivity regulation of the three cones' responses. 11Therefore, a chromatic adaptive transformation (CAT) method is adopted to address the perceptual mismatch between the actual scene and the displayed image.Several color appearance phenomena based on the HVS, such as the Hunt, Steven and Helson-Judd, and simultaneous contrast effects, can be predicted by the CAT method. 12,13Related Work Chromatic adaptation is the ability of the HVS to adjust to illumination changes and to preserve the color appearance of the object. 14It allows us to see stable object colors illuminated under a wide range of different illuminations.CATs are used in digital imaging and color science to model the described mechanism of the HVS.They provide a means to transform color values under a source illumination into color values under a target illumination.
A standard model to compute the transformation from one illumination to another is the diagonal von Kries adaptation model. 15If ðR; G; BÞ denotes a color value under a source illumination, then the von Kries model states that we can model the same color value under a target illumination as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 3 2 6 ; 6 3 5 where c R, c G, and c B represent scaling coefficients for the color channels.These scaling coefficients are generally the ratios of target illumination ðR t ; G t ; B t Þ and source illumination and c B ¼ B t ∕B s .CATs, however, there is a difference in the color space in which this scaling occurs.R t0 , G t0 , B t0 and R s0 , G s0 , B s0 are the final target illumination and the initial input source illumination.
The obvious choice is the color space in which the image is initially described, such as the sRGB color space.This process is straightforward as no additional transformations of the color space are required.Other commonly used color spaces are derived as linear transformations of the XYZ tristimulus space.Some of these alternatives are called sensor sharpening. 16Color-matching functions of the derived color spaces tend to produce sharper, narrower peaks, more appropriate for the von Kries model.There are several basic models for transforming color values in a derived color space including XYZ, Bradford, Sharp, and CMCCAT2000.All of the described transformations implement the diagonal von Kries model-to-model illuminant change.

Proposed Method
The input data ½fðx; yÞ for the proposed method has a CIE tristimulus value X, Y and Z) in absolute luminance units.The absolute luminance Y of the image data is necessary to predict the various luminance-dependent phenomena such as the Hunt and the Stevens effects.An HDR image input is typically a floating point RGB image with a linear absolute luminance.The captured R s , G s , and B s values are converted to the CIE XYZ color space using: 9,13,17 0.4124 0.2127 0.0193 0.3576 0.7152 0.1192 0.1805 0.0722 0.9504 where the R s , G s , and B s values refer to the sRGB of the input image.

Tone-Mapping Method
Conventional methods for tone mapping are generally performed using spatial filters such as Gaussian and bilateral.
Then the illumination components and the local adaptation level are estimated.However, methods based on spatial filters are known to yield inferior color constancy. 14,18For example, a halo artifact appears in the resulting image.To avoid color distortions, tone mapping is implemented without the gray world assumption.Therefore, the R, G, and B components of the color space are used in the previous tonemapping work as mentioned in Ref. 19 and consequently, the tone-mapping method cannot address the various HVSbased luminance-dependent phenomena. 1Therefore, the CIEXYZ tristimulus values (X, Y, and Z) are used in the proposed method.The tone-mapping method used in the previous one as the global tone-mapping method leads to color distortion.Moreover, it is used to render the color in the given image using just a global color rendering operator.However, the proposed method has a different tone-mapping operator compared to the previous one.Thus, the linearity is the important property in the tone mapping process, now that it is capable of addressing the one-to-one correspondence between the rendered image and the real world scene.In the effort, consequentially, the proposed method is of near linearity compared with the previous one, even though a nonlinear equation is used to control the dynamic range in the given image.In addition, the proposed color rendering operator is used to represent the local detailed variance information in the given image, adding a local standard deviation, which was not in the previous one.The input image is obtained using a nonlinear power function for the luminance (Y component) of the CIEXYZ tristimulus values.The nonlinear power function is adopted to control the dynamic range and to remove the halo artifact instead of using the spatial filter to estimate a white point.Therefore, for luminance Y½f Y ðx; yÞ, the tone cover of the achromatic component is defined as follows: where L out ðx 0 ; y 0 Þ is the output image for the absolute luminance Y and α is the gamma coefficient to control the dynamic range.The product of Eq. ( 3) is used by the tone-mapping function to preserve luminance, expressed in cd∕m 2 .This involves a linear interpolation between the chromatic and the corresponding achromatic colors.The proposed tone-mapping method is described using the input image f i ðx 0 ; y 0 Þ as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 6 3 ; 2 9 2 where C out;i ðx 0 ; y 0 Þ, i ¼ fX; Y; Zg are the resulting tonemapped images of the X, Y, and Z components, expressed in cd∕m 2 and the local standard deviation ½σ i ðx; yÞ is used to describe changes in the entire image The local standard deviation is based on the 3 × 3 mask for each R, G, and B channel in the given image, and it is used to compensate the global operator by incorporating the local variance information.As mentioned, the conventional color rendering method with linearity is based on the gray world assumption.Hence, the resultant image leads to poor color constancy such as halos, graying-out, and dominated color, which is caused by increasing the hue values in the resulting image.
Then the color cast is found in the entire resulting image after correcting the color.Even though the nonlinearity is used to correct the color so as to avoid these problems in the conventional methods, the objective of Eq. ( 4) is to ensure near linearity but is not a linear equation.It is modified from a portion of the tone-mapping method described in Ref. 1.The characteristic curve is shown in Fig. 1.The resulting image has linearity comparable, approximately, to that of the previous tone-mapping method.

Proposed Chromatic-Adaptation Transformation
The tone-mapping method was discussed in the previous subsection.To address the perceptual mismatches between the original scene and displayed images from the resulting image based on Eq. ( 4), the chromatic-adaptation transform has been adopted in the proposed method.The color appearance model (CAM) is used to estimate the cone response using the cone-response function in Ref. 9.However, there is a problem in the resulting image related to a dominated color.Therefore, a modified CAM model is introduced to address the mismatch between the displayed image and the rendered image based on HVP.The proposed transformation is obtained as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 5 ; 6 3 ; 5 8 8 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 6 3 ; 5 where R w , G w , B w , and X w , Y w , Z w values represent the corresponding R, G, B and C out;X , C out;Y , C out;Z values for the white point, respectively.From Eq. ( 5), three cones' responses based on the HVS are estimated as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 7 ; 6 3 ; E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 8 ; 6 3 ; E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 9 ; 6 3 ; where E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 0 ; 6 3 ; 3 4 6 L a ¼ L w ∕5; E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 1 ; E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 2 ; 6 3 ; E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 3 ; 6 3 ; 2 7 6 D oos ¼ 0.3 where D oos is the incomplete white point adaptation factor and is computed as the function of the adaptation luminance L a (20% of the adaptation white).Fl (luminance surround factor) is used to address the diverse variety of the actual scene.R c , G c , and B c values are the chromatically adapted cone responses from applying D oos .In the proposed method, X w , Y w , and Z w values can be incorporated as user parameters or can be estimated from the given data, as shown in Table 1.R rw , G rw , B rw and X rw , Y rw , and Z rw values are reference whites in the white condition, respectively.From R c , G c , and B c values, R 0 , G 0 , and B 0 values are calculated as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 4 ; 6 3 ; 1 0 5 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 6 ; 3 2 6 ; 3 1 8 where M HPE is a single transform matrix from the sharpened cone responses to the Hunt-Pointer-Estevez cone responses. 20n Eq. ( 14), R 0 ; G 0 , and B 0 values are the chromaticadaptation values that are obtained using the inverse M −1 CAT02 transform.Subsequently, they are converted to the HPE space before performing the postadaptation nonlinear compression.The CIECAM02, such as CIECAM97, uses a certain color space in the chromatic-adaptation transform and another for computing the correlation between the perceptual attributes.CIECAM02 exhibits the best performance in predicting the chromatic adaptation of the image with some degree of sharpening.It should be noted that the CAT02 method also incorporates some degree of sharpening.In comparison, the usage of space closer to the cone fundamentals such as the HPE or Stockman-Sharpe provides improved predictions of the perceptual attribute.Blue constancy, a significant shortcoming in CIELAB, is considerably improved using a space closer to the cone fundamentals. 9

Experiment Results and Evaluations
As discussed in the previous section, the proposed method incorporates Eq. ( 4), a tone-mapping function, and the coneresponse function to correct the color.In this section, we discuss the difference between the proposed method and some existing methods with respect to visual quality.To demonstrate the feasibility of correcting the color, the results using a number of well-known images are presented.These images are available at the links. 21,22We also used the images captured with five standard illuminations (D, CWF, TL84, A, and UV) and stored in tiff format.The images from Figs. 2-4 present the resulting image for HDR images after performing color correction using the techniques of iCAM06, 9 3) is set through an empirical test.The parameter is increased from 0 to 1.When the parameter is set as α ¼ 0.2, the result has best performance in the given images and the resultant image is improved without color distortion; that is, the halo and dominating color problems are substantially reduced when compared with both (b) and (c), as shown in the resulting images (Figs.2-4).The veiling glare is also markedly reduced.Figure 5 illustrates why the resulting image of the previous one appeared to add a blue cast compared to Figs. 4(c) and Fig. 4(d).It is biased in the direction of the blue color as shown in the result compared to the proposed work.Five images with different illuminations (D, CWF, TL84, A, and UV) are used to evaluate the effect of the illumination condition.Figure 6 shows the resulting images using the five images with different illuminations, and the standard illumination tool box is used to obtain these images but not outside.Figure 7 presents the luminance surround factor distribution of these images in Eq. (11).Both axes are represented as a pixel number of the input image and the result of the luminance surround factor, respectively, as in Eq. (11).The value in the iCAM is defined with F ¼ 1 for an average surround, or F ¼ 0.8 for dim-and dark-surrounds, whereas the results in the proposed method [Eq.(11)] are of the S-shape curve compared with that of iCAM06.A wide variety of scenes in the given image is addressed using the proposed surround luminance factor, where the input information is the luminance level of the pixels in the given image.The maximum values are (a) 0.0971 for "A" illumination, (b) 0.0959 for that of "CWF" illumination, (c) 0.1010 for "D65" illumination, (d) 0.881 for "TL84" illumination, and (e) 0.0804 for "UV" illumination.Figure 8  Figure 9 shows the resulting image according to the parameters for Eq.(3). Figure 9(a) is the original image and it is captured under a "D65" standard illumination atmosphere.The color difference (Δu 0 v 0 ) between the captured image under standard illumination and the color reproduction image is introduced to evaluate the color reproduction  performance of the proposed method using CIE 1976 u 0 v 0 color space, 24 which is widely used for color reproduction evaluations.
Figure 10 and Table 2 display the results obtained from mean color difference for the iCAM06, the previous technique, 1 and the proposed method.In the resulting diagram, the indicators of the proposed method are of a lower mean color difference 25 compared with the conventional method.Based on the captured image under D65 standard illumination, the maximum difference between the original image and rendered image is 0.0192.The human eye perceives a color difference when the difference of the two separated color patches is Δu 0 v 0 ≥ 0.04. 26,27Therefore, this indicates an excellent performance for the experiment because the color difference is smaller than the visual minimum perceptible color difference.Figure 11 presents the resulting images of the proposed method.In each of image, (a) is the original image, (b) is obtained by using iCAM06, (c) is that of the previous one, and (d) is the resulting image of the proposed method.The results obtained were similar to those for the above-mentioned approaches.
To conduct a subjective evaluation, a psychophysical experiment was performed.A total of 15 observers with normal color vision participated in the test, and five different illumination images were used to access the color rendering algorithm that considered the color reproduction, brightness, and colorfulness.These images were captured under five different illumination conditions, as shown in Fig. 6.For the psychophysical experiment, a pair comparison method was used.The iCAM06, the previous one, and the proposed method were compared.Because HVS has a veiling glare limit in a white background, such as a lighted room, the psychophysical experiment was conducted in a dark room.The parameter used for each algorithm is fixed based on the value suggested in the research paper.Each observer judged a pair of images and assigned a "1" to the selected image and a "0" to the rejected image.In the case of a draw, "0.5" was assigned to each image.The scores were then added and transformed to the preference scores. 28Figure 12 and Table 3 show the result of the psychophysical experiment for the captured images under five different standard illuminations.The preference scores in the proposed method are higher than those of the conventional methods.Figure 13 and Table 4 show the result of the preference score for the 30 different images.The result is similar to that of Fig. 12 and Table 3.

Conclusion
In the natural world, eyes are confronted with a broad range of luminance.Eye adaptation is the mechanism that allows our eyes, by changing their sensitivity, to be responsive at varying illumination levels.HDR imaging is a technique that mimics the human eyes and readily captures a range of light intensities in multiexposure time.The HDR imaging technique, however, must address the issue of displaying HDR images on LDR display devices.Tone-mapping operators are the typical algorithms designed to produce visibility and the overall impression of the brightness, contrast, and color of HDR images on LDR display devices.Although several new tone-mapping operators have been proposed in recent years, the results of these operators have not matched those of the psychophysical experiments based on the HVS.The mismatches relate particularly to color as described in many of the research articles.Some of the conventional methods also include issues, such as halo artifacts, graying-out, and dominating color.
In this paper, a color-rendering method, comprised of tone-mapping and a chromatic-adaptation method based on an XYZ tristimulus color space, was proposed.The method addressed the existing problems in the conventional methods.The tone-mapping operator produced visibility and an overall impression of brightness, contrast, and color in the given HDR images when mapped onto LDR display devices.The tone-mapping resultant image was obtained based on     The luminance factor was used to manage a diverse variety of actual scenes instead of the average surround in iCAM06.
In the experiment of the gamut and color difference using the CIE xy color space, over 30 images and five different illumination images were used for evaluation.The results for the proposed method demonstrated that the dominated color was remarkably reduced throughout the image compared to the conventional method, regardless of the illumination.The color difference equation is presented to conduct quantitative evaluation for color reproduction in the given image with five different standard illuminations.In the evaluation of the color difference, most of resulting images in the proposed method are of lower values compared with the conventional method, except for that of "CWF" standard illumination image.In order to conduct a subjective evaluation, the psychophysical experiment is performed using five different standard illuminations and 30 different images.As a result, the preference scores of the proposed method are higher than that of the other methods.The proposed method shows a better performance than others through several evaluation techniques, such as color difference equation, psychophysical experiment, and so on.However, the proposed method will somewhat improve the performance in that the goal of tone mapping is to produce visibility and an overall impression of brightness, contrast, and color in a given HDR image when mapped onto relatively LDR displays or printers based on HVP.In the near future, we will continue to investigate these problems.

Fig. 1
Fig. 1 Curve of characteristic based on Eq. (4) with (a) X value of tristimulus values, (b) Y value of tristimulus value, and (c) Z value of tristimulus values.

E
Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 the technique proposed in Ref. 1, and the proposed work.For each, (a) is the original image; (b) shows the resulting image after processing with iCAM06.The MATLAB codes of iCAM06 are publicly available 23 and the resulting images are obtained by using the parameters described in Ref. 9. The halos appear in the resulting image as the image adapts to the piecewise bilateral filter.Moreover, a red cast has been added to the entire resulting image because of the luminance factor in the coneresponse function of the chromatic-adaptation transform.In the result images of (c), as proposed in Ref. 1, halos are considerably reduced in the entire resulting image compared to (b).A dominating color appears in the entire resulting image.Because it has a curve of a nonlinear characteristic in part of the tone-mapping section, as shown in Ref. 1, the resulting image has a blue cast.Moreover, both (b) and (c) have a veiling glare similar to lamp lights despite the use of a nonlinear property and both chromatic and achromatic colors.The parameter (α) in Eq. (
is a quantitative evaluation using a gamut area [ICC3D version 1.2.9, copyright (c) 2002-2003 Gjovik University College].Figure 8(a) is the gamut area for "A" standard illumination, (b) "CWF" illumination, (c) "D65" illumination, (d) "TL84" illumination, and (e) "UV" illumination, and Symbol a+(or b+) and a− (or b−) are a*(or b*) positive and negative values of the CIELAB, respectively.The colors are distributed similarly in the gamut area.Therefore, the results indicate that the proposed method can affect the performance regardless of the illumination condition.

Figure 9 (
b) is the resulting image for parameter ∝¼ 0.1, Fig. 9(c) is the resulting image when setting parameter ∝¼ 0.2, Fig. 9(d) is that for ∝¼ 0.3, Fig. 9(e) is ∝¼ 0.4, and Fig. 9(f) is ∝¼ 0.5, respectively.The color changes take place in the entire resulting image when setting the parameter over ∝¼ 0.3.Figures 9(b) and 9(c) show a better performance than the other images.For selection of the best resulting image according to the parameter, 15 observers participated in the test.Most of the observers selected Fig. 9(c), ∝¼ 0.2, and that is the reason it is set as the parameter value in Figs.2-4.

Fig. 10
Fig. 10 Mean color difference (Δu 0 v 0 ) for the captured images under five different standard illuminations.
chromatic and achromatic colors (absolute luminance Y component).The resulting image, thereafter, was processed with the cone-response function wherein emphasis was placed on the HVP that addressed the mismatch between the original scene and the rendered image based on HVS.

Fig. 11
Fig. 11 Experimental resulting images using the proposed method: (a) the original image, (b) using iCAM06, (c) that of the previous one, and (d) the proposed method.

Fig. 12
Fig. 12 Preference scores for the captured images under five different standard illuminations.

Table 1
9xamples of the relative tristimulus values of the white points and their associated color temperatures T .9

Table 2
Mean color difference for the captured images under five difference standard illuminations.

Table 3
Preference scores for the captured images under five different standard illuminations.

Table 4
Preference scores for 30 different images.