Recently, high-fidelity color image reproduction has become increasingly important because of its many potential applications in textiles, medicine, digital archives, etc. To view objects under various illumination conditions, multispectral imaging has been extensively studied to estimate spectral reflectance of object surfaces.1, 2, 3, 4, 5, 6, 7 Multispectral images are usually acquired by trichromatic or monochrome cameras, accompanied by a set of color filters. The imaging process is often modeled by a linear system when the number of imaging channels is large. In recovery of spectral reflectance, Wiener estimation is deduced under the condition that the data conform to normal distribution.8 It works fairly well when the linearity condition is satisfied, and becomes a standard technique. However, it is likely that measured data are not in accordance to the normal distribution,2, 3 and the optoelectronic conversion function (OECF) of the camera is nonlinear.9 Nonlinearity usually degrades the estimation accuracy of linear methods. Adaptive methods, which use local statistics instead of global ones, provide feasible solutions for improving accuracy.2, 4 However, as adaptive methods usually need to recalculate the transform between responses and reflectance,4 they are computationally expensive and hence unsuitable for time-critical applications. An alternative way is to introduce nonlinear variables such as high-order polynomials. Hong, Luo, and Rhodes10 applied ordinary polynomial regression to predict colorimetric stimulus values from three-channel camera responses. The same technique has also been adopted in multispectral imaging.5 However, the extension of polynomial responses causes overfitting and collinearity problems when the number of imaging channels is large. Heikkinen 6 introduced regularized polynomial modeling methods and a more general regularization framework for robust reflectance estimation.
We propose a global method for spectral estimation based on polynomial extension of camera responses and partial least-squares (PLS) regression.11, 12 The PLS is implemented in an iterative manner; its dimension (or number of PLS components) is determined by the spectral error distribution. The accuracy of the PLS method is compared with Wiener estimation and also polynomial regressions solved by ordinary least squares (OLS) and regularized least squares (RLS).
Suppose that the continuous visible spectrum is uniformly sampled at (usually ) discrete wavelengths, and the number of imaging channels is . Let be the OECF of camera, be the reflectance of imaged object surface, and be the spectral responsivity of the imaging system, then camera response is computed asdenotes noise. When the camera behaves linearly, ; otherwise it can be represented by high-order polynomials.9
By ignoring , the linear method tries to find a transform matrix such that the estimate of reflectance can be computed ascan be calculated by Wiener estimation as4, 7 denotes the matrix transpose, and and are the autocorrelation matrices of reflectance and noise , respectively.4 and hereafter is numerically solved by singular value decomposition.13
Nonlinearity can degrade the performance of the linear reflectance estimation methods. It is natural to define a two-order polynomial response vector to deal with the nonlinearity due to OECF and non-Gaussian data distribution:is the ’th element of . As , has elements. By defining the polynomial response matrix as , the transform matrix can be solved under the OLS criterion as .
Alternatively, can also be computed by introducing a regularization (or penalization) term asis a regularization parameter and denotes the identity matrix. The overfitting problem can be diminished by the regularization term.6
Partial Least-Squares-Based Reflectance Estimation
As mentioned, the overfitting problem can occur in polynomial regression solved by OLS when the number of parameters in the mathematical model is greater than the number of dimensions of data variation. The overfitting problem is probably related to the increasing colinearity between the extended polynomial responses. In this regard, we propose to deal with it by dimensionality reduction using the PLS technique.11
The polynomial response matrix can be decomposed into a score matrix and a loading matrix , with being the number of PLS components, asis a residual matrix. Similarity, can also be decomposed into a score matrix and a loading matrix as is a residual matrix.
The goal of PLS is to extract the common structure between and by searching a projection such that the covariance between the score matrices and is maximized. In matrix form, this relationship is written asis the diagonal regression matrix.
The PLS algorithm is carried out in an iterative manner.12 To obtain an orthogonal score matrix , a weight matrix is introduced in the iterative procedure. Let be the iteration index, and , , , and be the ’th column vectors of matrices , , , and , respectively. Before starting the iteration, let and . Matrices and are then column centered and normalized so that each variable has zero mean and unit variance. Let and be any column of , then compute Eqs. 11, 12, 13, 14 iteratively:has not converged, return to Eq. 11, otherwise compute the value of , which is the ’th diagonal element of matrix , as , and compute the factor loading, which is the ’th column vector of , as .
The residual matrices and needed for the next iteration are calculated as
Note that Eqs. 15, 16 remove the variance associated with the obtained score and loading vectors before the next iteration. If , let and continue the iteration starting from Eq. 11; otherwise, stop the iteration and compute the PLS transform12.
As controls the number of iterations, its value is influential to PLS. If , and become zeros and PLS reduces to OLS; otherwise, if , the colinearity of matrix is reduced.12 In this work, is determined according to the spectral accuracy of the reflectance estimation, as is discussed in the following section.
In the multispectral imaging system, we used a monochrome digital camera (model Cool-SNAP HQ2, Roper Scientific Incorporated, Ottobrunn, Germany) with digitization and 16 narrowband filters ( half-width, product of Andover Company, Salem, New Hampshire) that uniformly cover the visible spectrum ranging from . The response of the camera deviates from linearity by less than 1%.14 As it is difficult to accurately acquire OECF at such a nonlinear level by imaging a number of gray samples, we consider it appropriate to treat it by extending the polynomial camera responses, as discussed in previous sections. We used 414 textile Pantone patches as the color targets, with half for training and half for testing. The surfaces of these patches contain weak textures and some gloss reflection, and hence are not ideal diffusers. The reflectance data of these patches were measured by the spectrophotometer GretagMacBeth (Grand Rapids, Michigan) 7000A; the 16-channel multispectral images were acquired by the imaging system. To reduce imaging noise, three sequential images were captured and averaged for the same scene, and the responses of each color sample were averaged in spatial areas with approximately . The transform between camera responses and reflectance was calculated from the training set and evaluated on the test set. The estimation accuracy was examined by spectral root-mean-square (rms)1 error and CIEDE2000 color difference15 error under CIE standard illuminants. We tried different randomization strategies for selecting training and test samples, and found that the estimation accuracies were quite close. Hence we only present the experimental results of the case where the odd numbered samples were used for training and the rest for testing.
The performance of the PLS method is compared with the OLS and RLS methods. For the RLS method, we set the regularization parameter , which approximately produces the minimum spectral rms error. For the PLS method, it is found that the spectral error is not very sensitive to the number of PLS component . The suitable value of is in the range from 40 to 60, and we adopted for computational efficiency. Table 1 gives the spectral and colorimetric errors of the PLS method, compared with the Wiener estimation, OLS, and RLS methods. It is clear that in terms of both colorimetric and spectral error metrics, the proposed method outperforms the Wiener estimation and OLS methods while being close to the RLS method. This is expected, as Wiener estimation cannot account for nonlinearity, and the OLS method has the inherent overfitting problem. The RLS method exhibits improved estimation accuracy through the introduction of the regularization term. By removing the common variance in the response and reflectance matrices, PLS also yields good accuracy.
Spectral rms errors and CIEDE2000 (ΔE00) errors of Wiener estimation, OLS, RLS, and PLS methods.
|Method||Spectral rms||ΔE00 under D65||ΔE00 under F2|
Due to the iterative nature of PLS, the computation time of the training procedure (calculation of ) is approximately proportional to . As the transform matrix is calculated before reflectance estimation, this computation time does not affect algorithm efficiency. The computation time of the test procedure is determined by the size of the matrix and is irrelevant to . The PLS method was implemented under the MATLAB® environment and run on a PC with an Intel Core 2 CPU at and with memory. For a multispectral image with , the PLS method costs about and can be much faster if programmed using C language. This indicates that the computational efficiency of the PLS method is acceptable to many practical applications.
We propose a method for estimating reflectance from multichannel camera responses based on high-order polynomials and partial least squares. The proposed method is capable of dealing with nonlinearity in the imaging process. The appropriate number of PLS components is determined based on spectral rms error distribution. In terms of spectral and colorimetric error metrics, the proposed technique is superior to Wiener estimation and polynomial regression solved by ordinary least squares, and is close to polynomial regression solved by regularized least squares.
We thank the reviewers for their comments that substantially improved this work. This work was supported by the NSF of China under grant 60778050.