1.

Introduction

Digital designers would benefit from tools that allow them to define custom color enhancements that are easy to use, share, edit, and do not require expert knowledge of color processing or statistical learning. A color enhancement is defined by a mapping from an original colorspace to another colorspace. For example, one color enhancement would be to transform all color values to pastel colors. As recently proposed,¹ one practical architecture for custom color enhancements is for a user to input a small set of example color transformations from which the complete enhancement’s color mapping can be estimated. Estimating the envisioned color-space mapping from only a few samples is a difficult estimation problem, but a solution that uses linear regression over a local neighborhood was shown to give promising results.² We expand on that work in this paper.

Three examples of user-defined sample color pairs are shown in Figs. 1, 2, 3 . In Fig. 1, the sample pairs are shown as vectors in the CIELab color space; each vector connects one of the 24 color patches of the standard Gretag Macbeth color chart photographed under a specific illuminant to the same color patch photographed under a different illuminant; more details are available in Section 6.1. An example of an image enhanced with a look-up-table (LUT) estimated from the cloudy-to-sunset sample pairs is shown in Fig. 2. Another example of color sample pairs is given in Fig. 3, which shows how 16 colors would be displayed using the Cinecolor film process; results with the Cinecolor enhancement are shown in Fig. 5. The goal for estimating a color enhancement from a small set of sample color pairs is to accurately capture the user’s envisioned color enhancement. More practically, we consider two metrics for the estimation: how well the estimated color enhancement accurately reproduces the user’s sample pairs and whether images enhanced with the estimated mapping are free of objectionable artifacts.

Fig. 1

Two examples are shown of sample pairs that describe a change of illuminant. Each sample pair is displayed as a vector in CIELab space that connects an input color to its enhanced color. The colors correspond to the colors of the standard 24 sample Gretag Macbeth color chart photographed under different illuminations.

Fig. 2

Photograph (a) was taken under an unknown illuminant, possibly a cloudy morning. Photograph (b) is a color-enhanced version of the left photograph, using a cloudy-to-sunset enhancement estimated from the cloudy-to-sunset sample pairs shown in Fig. 1. (Color online only.)

Fig. 3

Input and enhanced sample colors are shown that map original colors to how they would approximately appear in a Cinecolor film, according to the American Widescreen Museum Web site ³⁴. The Cinecolor color film process was used in the 1930s. Cinecolor used only two film colors (roughly described as red and cyan). (Color online only).

Fig. 5

The original image [shown in Row (1)] and the image transformed by the different Cinecolor LUT’s. Row (2): four nearest neighbors: (b) local linear regression, and (c) local ridge regression $\lambda =0.5$ . Row (3): smallest enclosing neighborhood: (d) local linear regression, and (e) local ridge regression $\lambda =0.5$ . Row (4): smallest enclosing inclusive neighborhood: (f) local linear regression, and (g) local ridge regression $\lambda =0.5$ . (Color online only).

Once estimated, the custom color enhancement can be stored as an International Color Consortium (ICC) profile. ICC profiles are the most widely adopted standard for characterizing and correcting color changes between devices, such as printers and monitors.^{3, 4, 5} The core of an ICC profile is a multidimensional LUT that spans the color space, such as a $17\times 17\times 17$ grid of points. The LUT defines how the colors on the grid are modified. The transformation of nongrid input colors are interpolated from the colors in the LUT. Color management modules that process ICC profiles are already implemented in many common hardware and software systems.⁴ Though developed and used to manage color between devices, ICC profiles provide a standardized and flexible architecture for defining any color transformation.

ICC profiles were standardized for color management. To estimate a LUT for color management of a device, one can expect to have on the order of a few hundred color sample pairs that span the color space and an underlying color transformation that is generally monotonic, though nonlinear. In contrast, to estimate a custom color enhancement LUT, it must be possible to estimate the LUT based on the order of 20 color sample pairs that do not necessarily span the color space. Additionally, the custom enhancement to be learned may be nonmonotonic as well as nonlinear. A first exploration of learning ICC profiles from sample pairs¹ showed that there were estimation trade-offs between oversmoothing the color enhancement and creating objectionable artifacts in enhanced images, such as unwanted false contours.

Building on previous work,² new neighborhood definitions are proposed for local linear and ridge regression in order to estimate custom color enhancements from a small set of color sample pairs. Ridge regression is a penalized form of linear regression that can reduce estimation variance. Both linear and ridge regression are flexible estimation methods when applied to local neighborhoods. We investigate adaptive neighborhood methods that attempt to enclose the points being estimated within a convex hull of neighborhood training samples. Extensive experimentation establishes the effectiveness of the new methods.

First, related research on color enhancements is reviewed in Section 2. Then, different estimation approaches are considered for estimating the ICC profile from the given color sample pairs in Section 3. A local learning approach gives the user both flexibility and control, but requires a local neighborhood to be defined. In Section 4, the literature on neighborhood definitions is reviewed, and we introduce the term enclosing neighborhood. New enclosing neighborhoods are proposed in Section 5. Experimental details are given in Section 6, followed by results in Section 7. The paper ends with a discussion on the future usage of ICC profiles and how enclosing neighborhoods may be advantageous for general learning problems.

2.

Related Research on Color Enhancements

Related to this work, Trussell proposed using a LUT to help consumers correct color by fitting a low-order polynomial to a small number of user-defined samples.⁶ Gatta ⁷ and Rizzi ⁸ proposed using spatially local LUT’s to approximate complicated color and image enhancements such as Retinex. For that application, no estimation is needed. Other researchers have worked on enhancing images using statistical learning techniques; for example, Hertzmann showed that learning could be used to enhance images by image analogies.⁹ Their work uses a pair of input images to define a transformation. Using input image pairs allows them to create spatial enhancements as well as color enhancements. Other researchers have developed methods to create custom color image enhancements by transforming the color palette of an original image based on the color palette of a single reference image.^{10, 11}

3.

Approaches to Estimation

Let $\mathbf{g}∊\mathit{Lab}$ be a gridpoint of a 3-D LUT of an ICC profile. Let $\widehat{\mathbf{y}}=\{\widehat{L},\widehat{a},\widehat{b}\}$ be the estimated enhanced CIELab color corresponding to $\mathbf{g}$ . Let $\{{\mathbf{x}}_i,{\mathbf{y}}_i\}$ for $i=1,\dots ,n$ be the user-defined sample color pairs, where ${\mathbf{x}}_i∊Lab$ and ${\mathbf{y}}_i∊\mathit{Lab}$ . The components of ${\mathbf{y}}_i$ are denoted ${\mathbf{y}}_i=\{L_i,a_i,b_i\}$ . If the user-defined color sample pairs are not originally in CIELab, then it is assumed that they are transformed to CIELab before processing. The problem is to estimate $\widehat{\mathbf{y}}$ for every gridpoint $\mathbf{g}$ of the LUT based on the given sample pairs $\{{\mathbf{x}}_i,{\mathbf{y}}_i\}$ .

A key issue behind the estimation is that the user’s small set of $n$ given color samples may not cover the full color space; thus, many gridpoints fall outside the convex hull spanned by the given color samples. It was shown in earlier work¹ that interpolative methods, such as tetrahedral linear interpolation or LIME,¹² clip colors outside the convex hull of the given color samples to the gamut defined by that convex hull. Instead, extrapolative methods, such as local linear regression,^{13, 14} must be used, at least for the outer gridpoints. Local linear regression has also been shown to work better than other regression methods for estimating LUTs for color management, including a neural net, polynomial regression, and splines.¹⁵

Local linear regression fits the least-squared error hyperplane to some local neighborhood of each gridpoint of the LUT. For a gridpoint $\mathbf{g}$ ,

Using too large a neighborhood can result in extrapolations that are too smooth and do not capture the the sense of the desired color transformation. Fitting a plane to a neighborhood that is too small results in a plane with a slope that is too steep, causing extrapolated colors to be grossly incorrect or clipped at the boundary of the display colorspace, which results in images with objectionable flat regions of clipped color. A related danger occurs when bright white is mapped to a nonwhite color, which can have the unintended effect that specular reflections and highlights in the image become colored in a manner inconsistent with the perceived whitepoint for the image.¹ Though affecting few pixels, the eye is very sensitive to such deviations from the perceived neutral axis⁴ and such colored highlights appear unnatural.

In mock workflow simulations with custom color enhancements, the ability to fine-tune a particular region of the color space was important to designers. Compared to global regression surface-fitting techniques, such as neural networks, local learning methods enable designers to locally edit the colors of a transform while minimizing changes to other parts of the colorspace. Another concern with neural networks is overfitting the surface to the training set. In related work by Trussell ,⁶ color corrections were implemented by fitting low-order global polynomials to a few user supplied color sample pairs. Local learning allows more flexibility to define a more complicated or precise color enhancement.

Ridge regression is used to stabilize the estimation.^{13, 16} Ridge regression forms a hyperplane fit, as in Eq. ¹, but the ridge regression coefficients $\overline{\beta }$ minimize a penalized least-squares criteria, which discourages fits with steep slopes. For example, for the luminance plane the ridge regression coefficients solve

Higher-order polynomial fits or spline fits are also possible solutions to estimating the LUT. These smoother fits would have more continuous derivatives. Because colors are perceptually indistinguishable if they are close enough, a point discontinuity is unlikely to be noticed, and we question the importance of continuous derivatives for this application. However, the slower variation of a higher-order polynomial fit could be an advantage over the disjoint fit formed by local linear regression. The disadvantages of such higher-order fits are an increased difficulty for the designer to fine-tune the enhancement, and potentially wild nonlinear estimates for colors that fall outside the convex hull of the training samples.

4.

Research into Neighborhoods

Local learning, such as local linear regression, requires a definition of a local neighborhood for each test point. For this application, each of the unknown gridpoints in the LUT will, in turn, be considered a test point. A common neighborhood choice is to use the $k$ nearest neighbors as defined by Euclidean distance, or as defined by some locally adapted distance.^{17, 18, 19, 20, 21, 22, 23, 24} In practice, it is common to choose $k$ by cross-validation.¹³

The challenge is to do well on average with a small set of training data. On the basis of experiments reported in Ref. 2 we hypothesized that using a neighborhood that encloses a test point in the convex hull of the neighborhood points when possible will work well on average. Define an enclosing neighborhood to be any set $\mathcal{J}$ of indices such that the gridpoint $\mathbf{g}={\sum }_{j∊\mathcal{J}}{w}_j{\mathbf{x}}_j$ where $\mathbf{w}∊{[0,1]}^{\Vert \mathcal{J}\Vert }$ is a weight vector over the set $\mathcal{J}$ subject to the constraint ${\sum }_{j∊\mathcal{J}}{w}_j=1$ . The neighborhood defined by Sibson²⁵ as the “natural neighbors” is an enclosing neighborhood (when possible). Although Sibson proposed natural neighbors with a specific generalized linear interpolation formula (called natural-neighbors interpolation), we consider them to be a promising neighborhood definition for more general learning tasks. To define Sibson’s natural neighbors, let $\mathcal{V}$ be the Voronoi tessellation of the complete set of training points $\left\{{\mathbf{x}}_i\right\}$ and test point $\mathbf{g}$ . The natural neighbors of $\mathbf{g}$ are defined to be those training points whose Voronoi cells are adjacent to the cell containing $\mathbf{g}$ .

Natural neighbors has been reported to be an accurate method for linear interpolation in two and three dimensions.²⁵ Unfortunately, to find the natural neighbors the entire Voronoi tessellation must be computed, which is computationally problematic as the dimension rises.^{26, 27} For 3-D color problems, such as the application considered in this work, natural neighbors are a feasible solution.

Sibson’s local coordinates property of the natural neighbors²⁵ proves that the natural neighbors form an enclosing neighborhood when possible. The local coordinates property establishes that, for a point $\mathbf{g}$ inside the convex hull of the entire training set, a weight vector $\lambda \left(\mathbf{g}\right)$ exists given $\mathbf{g}$ ’s natural neighbors $\mathcal{N}$ such that

Other neighborhoods have been defined based on spatial relationships between sample points.^{28, 29} For example, defining a test point’s neighbors as all Gabriel neighbors from the training set has been investigated^{26, 24} (pg. 90). Decision trees can be viewed as an adaptive neighborhood definition where the neighborhood is chosen to be (typically) a refined hyperrectangle or half-space that minimizes some empirical risk. A variation uses the decision tree to restrict the $k$ nearest-neighbor search to only those samples in the same branch of a learned decision tree.³⁰

5.

Enclosing Neighborhood Definitions

In this section, new enclosing neighborhood definitions are proposed. It is useful to define a distance to the enclosure of the neighborhood, which is the Euclidean distance between a test point and the convex hull of a set of training points. Given a gridpoint $\mathbf{g}$ , consider a set of neighborhood indices $\mathcal{J}$ . Let the distance to the enclosure of the neighborhood $\mathcal{J}$ about $\mathbf{g}$ be denoted $D(\mathbf{g},\mathcal{J})$ , and defined

Fig. 4

The four enclosing neighborhoods are compared. Image (a) shows the smallest enclosing neighborhood. Here, the point $x_6$ was needed to make it an enclosing neighborhood. Points $x_4$ and $x_5$ did not decrease the distance to enclosure after including $x_3$ and are thus excluded. Image (b) shows the smallest enclosing inclusive neighborhood, which includes $x_4$ and $x_5$ , as they are nearer than the furthest neighbor. Image (c) shows the natural neighbors marked as squares $(x_1,x_2,x_3,x_4,x_6,x_7)$ and the natural neighbors inclusive neighborhood, which additionally includes $x_5$ .

Natural neighbors was defined in Section 4. Here, a variant is proposed that may lead to lower estimation variance: Natural neighbors inclusive neighborhood. Let $\mathcal{N}$ be the set of indices of the natural neighbors for a given gridpoint $\mathbf{g}$ . The natural neighbors inclusive neighborhood includes any sample point ${\mathbf{x}}_j$ that is closer to the gridpoint than the furthest natural neighbor. That is, ${\mathbf{x}}_j$ is included in the neighborhood if

For the experiments, the following additional neighborhood definitions are compared: Four nearest neighbors: This neighborhood consists of the four sample points closest to the gridpoint $\mathbf{g}$ , which is the minimium needed to solve for linear regression coefficients (assuming all the sample points are in general position). All-but-one neighbors. Smoother interpolations should be achieved by regressing over larger neighborhoods. The all-but-one neighbors include all the training samples except the furthest training sample from the gridpoint $\mathbf{g}$ .

6.

Experiments

Neighborhoods are formed as per the descriptions in Section 5, where “nearest neighbors” are computed with respect to Euclidean distance in CIELab, and any distance or norm is computed in CIELab color space. If a neighborhood consisted of less than four neighbors, then nearest neighbors were added to make a minimum of four neighbors (to ensure stable regression). For each neighborhood method, a 3-D LUT is formed using local ridge regression with smoothing parameters $\lambda ∊\{0,.5,1\}$ . Ridge regression with $\lambda =0$ is equivalent to linear regression. The set of gridpoints $\mathcal{G}$ formed a $21\times 21\times 21$ LUT in CIELab color space with gridpoints spaced five units apart for the $L$ channel and spaced $10\text{units}$ apart for the $a$ and $b$ channels. All colors were either originally described in CIELab or originally described as RGB samples and transformed to CIELab using the standard sRGB-to-CIELab transformation with the default white point of D65.

For each of the different neighborhood definitions, a LUT was generated by estimating the corresponding output color $\widehat{\mathbf{y}}$ using local ridge regression for each gridpoint $\mathbf{g}∊\mathcal{G}$ . Then, each of the different LUTs was used to enhance the test set of images. The ICC standard does not constrain which estimation method is used for the LUT interpolation; the experiments used trilinear interpolation. Trilinear interpolation is a standard method for interpolating profiles³¹ and is a 3-D version of the common bilinear interpolation. Code for trilinear interpolation is available in Ref. 32 and is implemented by the Matlab function interpn.³³ Recently, it has been shown that trilinear interpolation weights the vertices of the LUT cell with weights that have the maximum entropy out of all solutions that satisfy the linear interpolation equations.¹²

The test set of images included 24 Kodak images from the Kodak Photo CD PCD0992 and the 918 sample Chromix color chart (available at www.chromix.com). The Kodak images have been released by Kodak to the public domain and are $24\text{bit}$ RGB color natural images of a variety of scenes with $768\times 512\text{pixels}$ . We converted the images to CIELab using the default sRGB-to-CIELab formula. For each estimation method, the image quality of enhanced test images was compared and the error on the training sample pairs was calculated to measure the fidelity of the enhancement to the given color sample pairs. The experimental data and images are available at idl.ee.washington.edu/projects.php.

For each neighborhood definition, the estimated LUT are compared on their ability to accurately recreate the original sample pairs $\{{\mathbf{x}}_i,{\mathbf{y}}_i\}$ . Each estimated LUT is used with each ${\mathbf{x}}_i$ to obtain the estimated $\widehat{{\mathbf{y}}_i}$ , which is then compared to the true training sample output color ${\mathbf{y}}_i$ . The mean and median $\Delta E$ CIELab errors for each estimated LUT for the given training samples are reported in Table 1 . The $\Delta E$ numbers are calculated as per the original CIE formula (Ref. 5, pg. 80), where $\Delta E$ is Euclidean distance in the CIELab space. These errors are based on testing on the training samples and thus reward overfit solutions. However, this still is a useful metric of the fidelity of the enhancement to the given color sample pairs. Furthermore, the overfitting potential is limited, as some smoothing will have taken place when estimating the LUT gridpoints, and smoothing occurs again when the original sample points are interpolated based on the cell vertices of the LUT. Furthermore, any overfitting that does occur is likely to lead to false contours in the enhanced test images and would be counted negatively under image quality.

Table 1

Errors for the Cinecolor transform. Errors have been rounded for display.

7.

Results

The results show that the proposed architecture can be used to effectively learn the tested set of custom enhancements. Example results are shown in Figs. 5 and 6 . Using ridge regression over any of the enclosing neighborhoods produced consistently good results. In Fig. 5, the Cinecolor transformation is shown for different neighborhood choices (rows) and with linear or ridge regression (columns). The images in Row (2) of Fig. 5 use four nearest neighbors. The ridge regression on the right (c) successfully dampens the wild extrapolations seen on the left (b), but there are still unacceptable image quality problems on the red sweater. Objectionable artifacts occur often in the test set when the four nearest neighbors are used with any of the ridge regression parameter settings.

Fig. 6

The original image [shown in Row (1)] and the image transformed by the different purple-and-gold LUTs that were estimated with local ridge regression with $\lambda =0.5$ . Row (2): (b) four nearest neighbors, and (c) smallest enclosing neighborhood. Row (3): (d) smallest enclosing inclusive neighborhood, and (e) natural neighbors. Row (4): (f) natural neighbors inclusive, and (g) all-but-one neighbors.

In Rows (3) and (4) of Fig. 5, images are shown for the smallest enclosing neighborhood and the smallest enclosing inclusive neighborhood, respectively. These two neighborhoods yield small estimation error on the original color sample pairs, as reported in Table 2 . However, they occasionally suffer image quality problems when used with linear regression. The objectionable artifacts disappear when ridge regression is used. There were only two instances of objectionable false contouring in the test set that were not removed by using ridge regression; both instances occurred with the Cinecolor transform. Additionally, ridge regression did not remove mildly objectionable color distortion on the Kodak hat test image when transformed with D65-to-Soft-Incandescent.

Table 2

The mean and median errors averaged over all 20 transforms. Errors have been rounded for display.

The natural neighbors neighborhood generally achieved high image quality, but on occasion led to objectionable false contouring. When used with ridge regression $(\lambda =0.5)$ , no objectionable false contouring was seen with the natural neighbors enhanced images. However, the Kodak test image of brightly colored hats transformed with the D65-to-Soft-Incandescent enhancement using natural neighbors showed mildly objectionable color distortion on one of the hats, even with ridge regression. The larger neighborhood size of the natural neighbors inclusive neighborhood did not exhibit objectionable artifacts in the test set, even when used with linear regression.

Using the all-but-one neighborhood never resulted in objectionable image artifacts, but failed to accurately capture some of the more nonlinear color-space transformations. For example, the results of the purple-and-gold enhancement for each neighborhood with ridge regression $(\lambda =0.5)$ is shown in Fig. 6. The image (g) is the all-but-one neighborhood, and the colors appear oversmoothed compared to the enclosing neighborhoods. In particular, because the neutral color axis was mapped to itself in the reference samples, the the bikers’ shirts should appear white. The larger neighborhoods smooth the colors such that the shirts do not appear white. The oversmoothing of the colors is reflected in Table 3 , which gives the average error on the original color sample pairs for the purple-and-gold transform. For $\lambda =0.5$ , the all-but-one neighbors has twice as much error as the smallest enclosing neighborhood for reproducing the original sample color pairs. Four neighbors achieves the lowest error on the original sample pairs, but the image shows that four nearest neighbors fails to produce reasonable image quality when faced with the larger test set of colors that appear in the image.

Table 3

Errors for the purple-and-gold transform. Errors have been rounded for display.

Similarly, the four nearest neighbors performs well in terms of fidelity to the original color sample pairs for the Cinecolor transform for $\lambda =0.5$ , as shown in Table 1, but the four neighbor images in Row (2) of Fig. 5 contain objectionable artifacts (here, the appearance of gray mold on the red sweater). Table 2 shows fidelity to the given enhancement color sample pairs averaged over the 20 different transforms. At $\lambda =0.5$ , the smallest enclosing neighborhood performs best by the total mean error metric and is tied for best by the total median error metric. Overall, the combined goals of fidelity and image quality are best satisfied for the test set by the smallest enclosing neighborhood with ridge regression at $\lambda =0.5$ . Not shown in the figures or tables are the ridge regression results for smoothing parameter $\lambda =1$ . Those results are generally very similar to the results with $\lambda =0.5$ . CIELab errors are difficult to interpret quantitatively, but larger errors will generally correspond to greater perceptual errors.

Histograms of neighborhood size for two example enhancements are shown in Fig. 7 . On average, the smallest enclosing inclusive is smaller than the natural neighbors neighborhood. The natural neighbors inclusive neighborhood is often quite large relative to the other enclosing neighborhoods and that explains the relatively color-smoothed images produced with this neighborhood.

Fig. 7

Histograms show the frequency of each size neighborhood over the 9261 gridpoints for (a) the Cinecolor transform, and (b) the cloudy-to-sunset transform.

8.

Discussion

In earlier work, we proposed the use of ICC profiles to capture custom color enhancements based on a small set of color sample pairs.¹ In this work, we have established that ridge regression over enclosing neighborhoods will produce reasonably accurate transforms with consistently good image quality and without the need for training (or cross-validation). In particular, new definitions of enclosing neighborhoods provide the most faithful color enhancements while rarely producing objectionable artifacts for enhancements where a smooth enhanced image could be expected.

ICC profiles are a flexible standard that has penetrated design software and hardware. This makes them an ideal vehicle for color processing far beyond the original intent of color management. In this work, we focused on how to learn an enhancement from an arbitrary small set of color sample pairs. As suggested in work by Gatta ⁷ and Rizzi ⁸ complex color transforming functions or programs can be approximated and implemented as ICC profiles. An open problem with color-space transformations of colors described as 3-D vectors is that they do not take into account semantic information about image content and thus face the problem of metamers. For example, regions of blue sky, blue jeans, and blue water may all have the same color value, but their physical nature is different and how they reflect incoming light spectra is different.

Another limitation of ICC profiles is the lack of spatial dependence. An ICC profile can be applied to segmented parts of an image, but the color transform itself ignores spatial information. This limits the ability to implement textural or spatial color enhancements. Color management scientists know that spatial effects on color perception are important and have proposed color appearance models.⁵ We ponder what a general spatial color transforming architecture would be that would have the flexibility and simplicity of the ICC profiles.