1 April 2010 Estimating reflectance from multispectral camera responses based on partial least-squares regression
Author Affiliations +
Abstract
In multispectral imaging systems, the accuracy of reflectance estimation can be degraded by the nonlinearity in imaging process, which is due to non-Gaussian distribution of the data and nonlinear optoelectronic conversion function of the camera. To deal with nonlinearity, we propose to extend camera responses by high-order polynomials and reduce the overfitting problem by partial least-squares (PLS) regression. Experiment shows that, in terms of both spectral and colorimetric error metrics, the proposed method performs better than Wiener estimation and ordinary polynomial regression, and is similar to polynomial regression with regularization.
Shen, Wan, and Zhang: Estimating reflectance from multispectral camera responses based on partial least-squares regression

1.

Introduction

Recently, high-fidelity color image reproduction has become increasingly important because of its many potential applications in textiles, medicine, digital archives, etc. To view objects under various illumination conditions, multispectral imaging has been extensively studied to estimate spectral reflectance of object surfaces.1, 2, 3, 4, 5, 6, 7 Multispectral images are usually acquired by trichromatic or monochrome cameras, accompanied by a set of color filters. The imaging process is often modeled by a linear system when the number of imaging channels is large. In recovery of spectral reflectance, Wiener estimation is deduced under the condition that the data conform to normal distribution.8 It works fairly well when the linearity condition is satisfied, and becomes a standard technique. However, it is likely that measured data are not in accordance to the normal distribution,2, 3 and the optoelectronic conversion function (OECF) of the camera is nonlinear.9 Nonlinearity usually degrades the estimation accuracy of linear methods. Adaptive methods, which use local statistics instead of global ones, provide feasible solutions for improving accuracy.2, 4 However, as adaptive methods usually need to recalculate the transform between responses and reflectance,4 they are computationally expensive and hence unsuitable for time-critical applications. An alternative way is to introduce nonlinear variables such as high-order polynomials. Hong, Luo, and Rhodes10 applied ordinary polynomial regression to predict colorimetric stimulus values from three-channel camera responses. The same technique has also been adopted in multispectral imaging.5 However, the extension of polynomial responses causes overfitting and collinearity problems when the number of imaging channels is large. Heikkinen 6 introduced regularized polynomial modeling methods and a more general regularization framework for robust reflectance estimation.

We propose a global method for spectral estimation based on polynomial extension of camera responses and partial least-squares (PLS) regression.11, 12 The PLS is implemented in an iterative manner; its dimension (or number of PLS components) is determined by the spectral error distribution. The accuracy of the PLS method is compared with Wiener estimation and also polynomial regressions solved by ordinary least squares (OLS) and regularized least squares (RLS).

2.

Previous Methods

Suppose that the continuous visible spectrum is uniformly sampled at N (usually N=31 ) discrete wavelengths, and the number of imaging channels is C=16 . Let F() be the OECF of camera, rRN×1 be the reflectance of imaged object surface, and MRC×N be the spectral responsivity of the imaging system, then camera response uRC×1 is computed as

1

u=F(Mr+n),
where nRC×1 denotes noise. When the camera behaves linearly, F(x)=x ; otherwise it can be represented by high-order polynomials.9

By ignoring F() , the linear method tries to find a transform matrix WRN×C such that the estimate of reflectance can be computed as

2

r̂=Wu.
W can be calculated by Wiener estimation as4, 7

3

WWE=KrMT(MKrMT+Kn)1,
where superscript T denotes the matrix transpose, and KrRN×N and KnRC×C are the autocorrelation matrices of reflectance r and noise n , respectively.

W can also be solved under the least-squares criterion. Let K be the number of training samples. We can construct reflectance matrix R=[r(1),r(2),,r(K)]RN×K and response matrix U=[u(1),u(2),,u(K)]RC×K , and then calculate the transform matrix as1, 10

4

W=RU+,
where superscript + denotes the matrix pseudoinverse. The pseudoinverse operator in Eq. 4 and hereafter is numerically solved by singular value decomposition.13

Nonlinearity can degrade the performance of the linear reflectance estimation methods. It is natural to define a two-order polynomial response vector ũRJ×1 to deal with the nonlinearity due to OECF and non-Gaussian data distribution:

5

ũ=[1,u1,,uc,u12,u1u2,,u1uc,u22,u2u3,,u2uc,,uc1uc,uc2]T,
where uc(1cC) is the c ’th element of u . As C=16 , ũ has J=153 elements. By defining the polynomial response matrix as Ũ=[ũ(1),ũ(2),,ũ(K)]RJ×K , the transform matrix W can be solved under the OLS criterion as

6

WOLS=RŨ+,
and reflectance is predicted as r̂=WOLSũ .

Alternatively, W can also be computed by introducing a regularization (or penalization) term as

7

WRLS=(RŨT)(ŨŨT+ηI)1,
where η is a regularization parameter and I denotes the identity matrix. The overfitting problem can be diminished by the regularization term.6

3.

Partial Least-Squares-Based Reflectance Estimation

As mentioned, the overfitting problem can occur in polynomial regression solved by OLS when the number of parameters in the mathematical model is greater than the number of dimensions of data variation. The overfitting problem is probably related to the increasing colinearity between the extended polynomial responses. In this regard, we propose to deal with it by dimensionality reduction using the PLS technique.11

The polynomial response matrix Ũ can be decomposed into a score matrix TRK×L and a loading matrix PRJ×L , with L being the number of PLS components, as

8

ŨT=TPT+E,
where ERK×J is a residual matrix. Similarity, R can also be decomposed into a score matrix DRK×L and a loading matrix QRN×L as

9

RT=DQT+F,
where FRK×N is a residual matrix.

The goal of PLS is to extract the common structure between ŨT and RT by searching a projection such that the covariance between the score matrices T and D is maximized. In matrix form, this relationship is written as

10

D=TB,
where BRL×L is the diagonal regression matrix.

The PLS algorithm is carried out in an iterative manner.12 To obtain an orthogonal score matrix T , a weight matrix GRJ×L is introduced in the iterative procedure. Let j be the iteration index, and g , t , q , and d be the j ’th column vectors of matrices G , T , Q , and D , respectively. Before starting the iteration, let E=UT and F=RT . Matrices E and F are then column centered and normalized so that each variable has zero mean and unit variance. Let j=1 and d be any column of F , then compute Eqs. 11, 12, 13, 14 iteratively:

11

g=ETdETd,

12

t=Eg,

13

q=FTtFTt,

14

d=Fq,
where ‖⋅‖ denotes the Euclidean norm. If t has not converged, return to Eq. 11, otherwise compute the value of b , which is the j ’th diagonal element of matrix B , as b=dTttTt , and compute the factor loading, which is the j ’th column vector of P , as p=ETttTt .

The residual matrices E and F needed for the next iteration are calculated as

15

E=EtpT,
and

16

F=FbtqT.

Note that Eqs. 15, 16 remove the variance associated with the obtained score and loading vectors before the next iteration. If j<L , let j=j+1 and continue the iteration starting from Eq. 11; otherwise, stop the iteration and compute the PLS transform12

17

WPLST=G(PTG)1(TTT)1TTRT,
based on which reflectance can be estimated as r̂=WPLSũ .

As L controls the number of iterations, its value is influential to PLS. If L=min(J,K) , E and F become zeros and PLS reduces to OLS; otherwise, if L<min(J,K) , the colinearity of matrix Ũ is reduced.12 In this work, L is determined according to the spectral accuracy of the reflectance estimation, as is discussed in the following section.

4.

Experiment

In the multispectral imaging system, we used a monochrome digital camera (model Cool-SNAP HQ2, Roper Scientific Incorporated, Ottobrunn, Germany) with 14-bit digitization and 16 narrowband filters ( 10nm half-width, product of Andover Company, Salem, New Hampshire) that uniformly cover the visible spectrum ranging from 400to700nm . The response of the camera deviates from linearity by less than 1%.14 As it is difficult to accurately acquire OECF at such a nonlinear level by imaging a number of gray samples, we consider it appropriate to treat it by extending the polynomial camera responses, as discussed in previous sections. We used 414 textile Pantone patches as the color targets, with half for training and half for testing. The surfaces of these patches contain weak textures and some gloss reflection, and hence are not ideal diffusers. The reflectance data of these patches were measured by the spectrophotometer GretagMacBeth (Grand Rapids, Michigan) 7000A; the 16-channel multispectral images were acquired by the imaging system. To reduce imaging noise, three sequential images were captured and averaged for the same scene, and the responses of each color sample were averaged in spatial areas with approximately 60×60pixels . The transform between camera responses and reflectance was calculated from the training set and evaluated on the test set. The estimation accuracy was examined by spectral root-mean-square (rms)1 error and CIEDE2000 color difference15 error under CIE standard illuminants. We tried different randomization strategies for selecting training and test samples, and found that the estimation accuracies were quite close. Hence we only present the experimental results of the case where the odd numbered samples were used for training and the rest for testing.

The performance of the PLS method is compared with the OLS and RLS methods. For the RLS method, we set the regularization parameter η=0.001 , which approximately produces the minimum spectral rms error. For the PLS method, it is found that the spectral error is not very sensitive to the number of PLS component L . The suitable value of L is in the range from 40 to 60, and we adopted L=40 for computational efficiency. Table 1 gives the spectral and colorimetric errors of the PLS method, compared with the Wiener estimation, OLS, and RLS methods. It is clear that in terms of both colorimetric and spectral error metrics, the proposed method outperforms the Wiener estimation and OLS methods while being close to the RLS method. This is expected, as Wiener estimation cannot account for nonlinearity, and the OLS method has the inherent overfitting problem. The RLS method exhibits improved estimation accuracy through the introduction of the regularization term. By removing the common variance in the response and reflectance matrices, PLS also yields good accuracy.

Table 1

Spectral rms errors and CIEDE2000 (ΔE00) errors of Wiener estimation, OLS, RLS, and PLS methods.

MethodSpectral rms ΔE00 under D65 ΔE00 under F2
MeanStandardMaxMeanStandardMaxMeanStandardMax
Wiener estimation0.01170.00650.04951.450.735.991.280.614.18
OLS0.01340.01400.09991.361.268.751.341.4310.33
RLS0.00790.00510.04450.920.745.570.840.603.96
PLS (proposed)0.00760.00430.03010.870.694.850.800.603.93

Due to the iterative nature of PLS, the computation time of the training procedure (calculation of WPLS ) is approximately proportional to L . As the transform matrix WPLS is calculated before reflectance estimation, this computation time does not affect algorithm efficiency. The computation time of the test procedure is determined by the size of the matrix WPLS and is irrelevant to L . The PLS method was implemented under the MATLAB® environment and run on a PC with an Intel Core 2 CPU at 1.86GHz and with 3-GB memory. For a multispectral image with 1392×1040pixels , the PLS method costs about 9s and can be much faster if programmed using C language. This indicates that the computational efficiency of the PLS method is acceptable to many practical applications.

5.

Conclusions

We propose a method for estimating reflectance from multichannel camera responses based on high-order polynomials and partial least squares. The proposed method is capable of dealing with nonlinearity in the imaging process. The appropriate number of PLS components is determined based on spectral rms error distribution. In terms of spectral and colorimetric error metrics, the proposed technique is superior to Wiener estimation and polynomial regression solved by ordinary least squares, and is close to polynomial regression solved by regularized least squares.

Acknowledgments

We thank the reviewers for their comments that substantially improved this work. This work was supported by the NSF of China under grant 60778050.

references

1.  H. L. Shen, J. H. Xin, and S. J. Shao, “Improved reflectance reconstruction for multispectral imaging by combining different techniques,” Opt. Express 15(9), 5531–5536 (2007). 10.1364/OE.15.005531 Google Scholar

2.  J. M. DiCarlo and B. A. Wandell, “Spectral estimation theory: beyond linear but before Bayesian,” J. Opt. Soc. Am. A 20(7), 1261–1270 (2003). 10.1364/JOSAA.20.001261 Google Scholar

3.  Y. Murakami, T. Obi, M. Yamaguchi, and N. Ohyama, “Nonlinear estimation of spectral reflectance based on Gaussian mixture distribution for color image reproduction,” Appl. Opt. 41(23), 4840–4847 (2002). 10.1364/AO.41.004840 Google Scholar

4.  H. L. Shen and J. H. Xin, “Colorimetric and spectral characterization of a color scanner using local statistics,” J. Imaging Sci. Technol. 48(4), 342–346 (2004). Google Scholar

5.  D. R. Connah and J. Y. Hardeberg, “Spectral recovery using polynomial models,” in Proc. SPIE 5667, 65–75 (2005). 10.1117/12.586315 Google Scholar

6.  V. Heikkinen, T. Jetsu, J. Parkkinen, M. Hauta-Kasari, T. Jaaskelainen, and S. D. Lee, “Regularized learning framework in the estimation of reflectance spectra from camera responses,” J. Opt. Soc. Am. A 24(9), 2673–2683 (2007). 10.1364/JOSAA.24.002673 Google Scholar

7.  N. Shimano, “Recovery of spectral reflectances of objects being imaged without prior knowledge,” IEEE Trans. Image Process. 15(7), 1848–1856 (2006). 10.1109/TIP.2006.877069 Google Scholar

8.  W. K. Pratt, Digital Imaging Processing, 2nd ed., Wiley, New York (1991). Google Scholar

9.  H. L. Shen, J. H. Xin, D. X. Yang, and D. W. Lou, “Estimation of optoelectronic conversion functions of imaging devices without using gray samples,” Color Res. Appl. 33(2), 135–141 (2008). 10.1002/col.20386 Google Scholar

10.  G. Hong, M. R. Luo, and P. A. Rhodes, “A study of digital camera colorimetric characterization based on polynomial modeling,” Color Res. Appl. 26(1), 76–84 (2001). 10.1002/1520-6378(200102)26:1<76::AID-COL8>3.0.CO;2-3 Google Scholar

11.  P. Geladi and B. R. Kowalski, “Partial least-squares regression: a tutorial,” Anal. Chim. Acta 185, 1–7 (1986). 10.1016/0003-2670(86)80028-9 Google Scholar

12.  L. H. Chiang, E. L. Russell, and R. D. Braatz, Fault Detection and Diagnosis in Industrial Systems, Springer, London (2001). Google Scholar

13.  W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C—the Art of Scientific Computing, 2d ed., Cambridge, UK (1992). Google Scholar

15.  M. R. Luo, G. Cui, and R. Rigg, “The development of the CIE 2000 colour-difference formula: CIEDE2000,” Color Res. Appl. 26(5), 340–350 (2001). 10.1002/col.1049 Google Scholar

© (2010) Society of Photo-Optical Instrumentation Engineers (SPIE)
Hui-Liang Shen, Hui-Jiang Wan, Zhe-Chao Zhang, "Estimating reflectance from multispectral camera responses based on partial least-squares regression," Journal of Electronic Imaging 19(2), 020501 (1 April 2010). https://doi.org/10.1117/1.3385782 . Submission:
JOURNAL ARTICLE
3 PAGES


SHARE
Back to Top