Spectral reflectance estimation of organic tissue for improved color correction of video-assisted surgery

Abstract. With video-assisted surgery devices becoming more common in all fields of diagnostics and therapy, the question of how well such systems are able to reproduce surface colors of organic tissue arises, especially in cases where a proper distinction of different kinds of tissue—not only through their texture but also through their color—might be crucial for the success of a surgery. Since modern devices are usually made of a highly efficient, multispectral LED light source in combination with some light-guiding structures and a digital camera system, an approach of optimizing these systems’ color reproduction properties based on the estimation of in-situ spectral reflectances is proposed. Following the International Organization for Standardization (ISO) standard procedure of colorimetric characterization, an initial color correction matrix was determined first by solving a linear least-mean-squares optimization problem for a small set of artificial color samples mapping the corresponding camera responses onto the samples’ tristimulus values. This initial matrix was then used as a starting point for a second, nonlinear optimization, which makes use of the estimated reflectance spectra in order to minimize the average squared color difference between the color corrected and the actually perceived tristimulus values of the individual organic tissue samples. Compared to the ISO standard, which only knows simple color patches to be used for the nonlinear optimization, a significant enhancement in the color reproduction of organic tissue could be confirmed with the newly proposed method being applied instead.

Spectral reflectance estimation of organic tissue for improved color correction of video-assisted surgery 1 Introduction Over the past few decades, minimally invasive surgery (MIS) techniques have become prevalent in various fields of surgical diagnostics and therapy, 1 offering many benefits over traditional surgery. These benefits include less tissue damage and, therefore, less pain for the patients, shorter recovery time as well as increased safety and precision. [2][3][4] Moreover, the use of MIS procedures instead of traditional open surgery, especially for routine operations, can lead to significantly reduced health care costs. 5,6 Researchers at Johns Hopkins University School of Medicine recently found that by increasing the use of laparoscopic techniques by 50%, US-American hospitals collectively could prevent thousands of postsurgical complications and save up to $280 million per year. 6 Currently, the majority of modern MIS systems are video based, giving surgeons an extended, in some cases, even three-dimensional view inside the patient's body revealing its anatomical topography and, therefore, allowing for better visual support during surgery. Making use of digital endoscopes, such systems enable surgeons to operate through a much smaller incision than would otherwise be required for traditional open surgery. At the same time, the surgeon's exact view of the surgical field can be provided to nurses or assisting surgeons with the same image quality by projecting the digital image of the MIS system on an additional monitor. This allows for better communication of surgical workflows and strategies and, therefore, more efficient collaboration in the operating room.
Despite the advantages of video-based MIS technologies and their increasing dissemination, there is still a lack of international standards and norms regarding the color reproduction properties of such fully digital visual systems. Modern endoscopic devices usually consist of a digital RGB camera system attached to a thin rigid or flexible tube, which can be inserted through small incisions or natural orifices (e.g., mouth or nostrils) into the patient's body. The endoscopic tube is designed in such a way that tiny surgical instruments can be inserted and used by the surgeon in order to reach out for and operate on the internal organs. It further contains a carefully optimized optical fiber and lens system. In combination with an external light source and the attached camera system, this ensures both the proper illumination of the organs under inspection and the correct transmission of the image from the objective lens to the capturing device, i.e., the digital camera.
However, image quality and color reproduction still remain a big issue for such medical applications, whichaccording to the authors' best knowledge-is addressed by only a few manufacturers because of missing obligatory standards in this field. Regarding other fields of application of digital camera systems, it is a known consensus that the issue of color reproduction has to be considered thoroughly to obtain qualitatively good and pleasant results. In this context, an interview-based investigation of the color image preference of surgeons revealed that a high acceptance of fully digital MIS devices could only be achieved if the experienced visual impressions resemble those usually perceived when using purely optical systems, i.e., systems they got used to over the years, which basically provide genuine colorimetric information. 7 Hence, even though a good surgeon under the assumption of sufficient image contrast might still efficiently perform a surgery, in case various organic tissues were represented in colors deviating from their actual ground truth, there is strong evidence that a proper color reproduction is desirable to achieve high user acceptance. Furthermore, it is the authors' belief that a colorimetrically genuine representation of the captured image data also provides an indispensable part for appropriate digital archiving and teaching purposes.
Assuming raw image capturing, the color reproduction of digital endoscopes and MIS devices is mainly influenced by the camera color filters, the optical transmission properties of the endoscopic tube, and the spectrum of the illuminating light source, which in modern solutions is formed by a highly efficient, multispectral LED light engine usually tuned to the emission of cool white light. In this context, perfect color reproduction means that the reproduced colors of, for example, inner organs would exactly match their direct perception under the same illumination. With this in mind, one can easily imagine that a more or less perfect color reproduction and, therefore, negligibly small colorimetric errors are crucial for the success of a surgery, where different kinds of tissues must be distinguished by the surgeon not only through their texture but also through their color. However, perfect color reproduction is usually not the case because, in general, the RGB-sensor of a digital camera system does not fulfill the required Luther-Ives condition, [8][9][10] i.e., the spectral response curves of the RGBsensor cannot be described as a linear combination of the eye cone response functions. As a consequence, more or less severe colorimetric errors occur when comparing the direct perception of the surgical field to its reproduction.
To correct for these systematic errors of digital cameras, various linear and nonlinear characterization approaches and methods have been proposed in the literature. [11][12][13][14][15][16][17][18][19][20] Despite their varying degree of complexity, they all have in common that, for a given illuminant, they try to minimize the perceived color differences between the reproduction of some test colors and the direct perception of the same test colors in a device-independant uniform color space (UCS). In practice, most manufacturers apply a simple linear leastmean-squares optimization based on some standardized color charts, as described by Hubel et al. 20 However, as one might expect, the reflectance properties of such printed color charts generally have nothing to do with those of real human tissue a surgeon is confronted with in the operating room and, therefore, such an optimization can only be considered as a rough estimate rather than a colorimetrically proper characterization. A significant improvement could only be achieved if the in-situ reflectance properties of the human tissue under inspection were known, which is usually not the case. The aim of this paper, therefore, is to provide an easy-to-implement reconstruction approach based on the Wiener filter 21 that estimates the spectral reflectances of organic tissue from a series of images captured under varying LED illumination. It should be further shown that once the in-situ spectral reflectances are known, significantly improved color correction is feasible.
The paper is organized as follows. In Sec. 2, the applied Wiener-filter approach for estimating the spectral reflectances is introduced. Section 3 gives an overview of the experimental setup, whereas in Sec. 4, some results of the spectral reflectance estimation are presented. In Sec. 5, a method based on in-situ measured reflectance spectra is discussed, which attempts to enhance the color correction properties of video-assisted MIS devices. Finally, the paper closes in Sec. 6 with some concluding remarks and an outlook on future research intentions.

Wiener-Filter Estimation Approach
Estimating in-situ spectral reflectances from digital camera responses of a modern endoscope generally is an ill-posed problem 22,23 since the dimensionality of the camera signals is much smaller than the number of spectral components that should be reconstructed. In the literature, several strategies have been proposed to tackle this problem making use of a priori knowledge of the underlying acquisition process. 12,[22][23][24][25][26][27][28][29][30][31][32][33][34][35][36] In this context, one of the most important and widely used reconstruction methods is the so-called Wiener estimation, which is also the method of choice in the present work. Despite its simplicity, the Wiener estimation approach has been proven to provide accurate results regarding the spectral reconstruction of nonflourescent natural reflectances. [22][23][24][25][26] Assuming a normal distribution and statistic independence of reflectances and system noise, the reflectanceestimating Wiener Filter, which is applied to the camera responses, minimizes the mean-squared error (MSE) between the estimated and the actual reflectance of a surface color. Basically, the responses c ði;jÞ ¼ ½c T of a linear trichromatic RGB camera system at a pixel location ði; jÞ on the camera sensor are given by (1) where S ¼ ðs T R ; s T G ; s T B Þ T is a 3 × n dimensional matrix containing the spectral sensitivities s R , s G , and s B of the digital camera as row vectors, DðxÞ is an operator that transforms a vector x into a diagonal matrix with the vector entries x 1 ; x 2 ; : : : ; x n forming the diagonal, l defines the spectral power distribution (SPD) of the illuminant, r ði;jÞ is a n × 1 dimensional column vector representing the spectral reflectance of the captured surface, and n ði;jÞ ¼ ½n ði;jÞ R ; n ði;jÞ G ; n ði;jÞ B T denotes the additive noise of the three RGB channels. Furthermore, the dimension variable n defines the spectral sampling (e.g., n ¼ 401 for spectra from 380 to 780 nm gives a 1 nm sampling) and δ ði;jÞ is some scaling factor depending on the emitted radiance of the illuminant and the measurement geometry, which is determined by the distance and angle of the object's surface with respect to the camera and illuminant.
For the sake of simplicity, the pixel position indices ði; jÞ are omitted in the following and Eq. (1) can be rewritten as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 3 2 6 ; 1 5 9 c ¼ Λr þ n; (2) where Λ ¼ δSDðlÞ is the combined lighting-sensitivity matrix of the system. In order to solve the above equation and to provide an accurate estimater for the actual reflectance, a Wiener filter matrix W is introduced to be applied to the RGB camera responses, which reads According to Refs. 23-25 and 37, the above reflectanceestimating Wiener filter matrix is defined in such a way that it minimizes the MSE between the estimated and the actual reflectance and, therefore, can be derived as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 6 3 ; 6 8 where K rr and K nn are the n × n and 3 × 3 autocorrelation matrices of reflectance spectra and system noise, respectively. For this study, the noises of different color channels are assumed to be independent, which means that K nn is diagonal. Further details on the derivation of Eq. (4) are given in Appendix.

Experimental Setup
Lacking a real MIS device equipped with a multispectral LED light source, we decided to set up an experimental environment, as shown in Figs. 1(a) and 1(b), which simulates the surgical workflow using a commercially available single lens reflex (SLR) camera and a customized six-channel LED light engine. Here, the abdomen or thorax of a patient is abstracted by a 40 cm × 40 cm × 60 cm wooden box with diffusely reflecting white painted walls on the inside in order to guarantee homogeneous illumination of the organic tissue under inspection. Please keep in mind that this setup should only provide a first proof-of-concept rather than mirroring reality. As can be seen from Fig. 1(a), the illumination is realized from above by a six-channel (red, deep red, green, blue, warm-white, and cool-white) edge-LED arrangement consisting of multicolor LED stripes being installed on all four sides of the box's lid. Several diffusing panels covering the LEDs and forming the bottom of the lid guarantee diffuse and homogeneous light emission into the artificial abdomen/ thorax. An aperture ring with a diameter of 10 cm embedded in the center of the lid together with a specially designed fixture allows for an easy insertion and alignment of different measurement devices. Furthermore, each LED channel can be controlled separately via pulse-width modulated (PWM) dimming and the corresponding normalized SPDs are illustrated in Fig. 2. They were measured in 0 deg ∕0 deg geometry using a white reflectance standard placed centrally inside the experimental box in combination with an Instrument Systems CAS 140CT array spectrometer connected to a TOP 200 radiance measuring head, as shown in Fig. 1 During the actual experiment, i.e., the image capturing and subsequent spectral reconstruction of organic tissue reflectances, the spectrometer head is removed from the box and a Canon 750D SLR digital camera is inserted at exactly the same position with its objective facing perpendicularly down toward the test object being placed right beneath the camera in the center of the bottom of the experimental box. Instead of human organs, freshly butchered pig Fig. 1 Illustrations of the artificial abdomen/thorax used in the experiments. The surgical workflow of a video-assisted MIS device should be simulated using a commercially available SLR camera and a customized six-channel PWM-controlled LED light engine. The artificial abdomen/thorax is formed by a 40 cm × 40 cm × 60 cm wooden box with diffusely reflecting white painted walls on the inside in order to guarantee homogeneous illumination. (a) Schematic representation of the experimental setup. The organic tissue sample is placed inside the box right beneath the measurement device, which is kept in position by a specially designed fixture. The illumination is realized using a multicolor edge-LED arrangement covered by diffusing panels and installed on all four sides of the lid. (b) Image showing the experimental setup as it can be found in our laboratory. Here, the radiance measuring head of the employed array spectrometer is attached to the fixture facing down toward the homogeneously illuminated test sample inside the box. Measurements were always performed in this 0 deg∕0 deg geometry.
Journal of Electronic Imaging 053012-3 Sep∕Oct 2018 • Vol. 27 (5) and cattle organs were used as test objects to investigate the performance of the reconstruction algorithm. Due to biological similarity, it is supposed that results can easily be transferred to MIS applications involving human tissue. As mentioned in Sec. 2, an easy-to-implement Wiener filter approach is used to estimate the spectral reflectances of the different test organs from the digital RGB camera responses of the Canon 750D. One problem that arises with this estimation method is that more than three camera responses are usually needed to obtain high accuracy. 38 This is because of the extremely ill-posed nature of the calculation when mapping only three values R, G, and B to reflectance intensities at a few hundred different wavelengths (e.g., 401 different wavelengths from 380 to 780 nm if 1 nm sampling is applied). In order to overcome this problem, we combine multiple captures of the test organs illuminated by different LED SPDs [six in our case, see Fig. 2]. As stated in Ref. 39, this allows us to significantly increase the effective dimension and, as a result, make a more accurate estimate of the spectral reflectances without the need of additional constraining assumptions on their properties (positivity, boundedness, and smoothness must be fulfilled for naturally occurring reflectance spectra).
Hence, the reflectance-estimating Wiener filter of Eq. (4) needs to be adjusted accordingly. Considering the six different illumination conditions l 1 ; : : : ; l 6 of our experimental box, the combined lighting-sensitivity matrix Λ must be rewritten as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 5 ; 6 3 ; 1 9 1 Λ ¼ δ½Dðl 1 ÞS T ; : : : ; Dðl 6 ÞS T T ; (5) which is now 18 × n dimensional. The corresponding 18 × 1 dimensional camera response vector c is consequently given by E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 6 3 ; 1 2 7 c ¼ ðc T 1 ; : : : ; c T 6 Þ T ; and contains the dark current corrected RGB camera values of the organic tissue surface captured under the six different LED SPDs. Concerning the computation of the autocorrelation matrix of reflectance spectra K rr , a database consisting of more than 300 different spectral reflectances of organic animal tissue (mostly pig and cattle organs) previously measured in our laboratory has been used. Last but not least, the noise autocorrelation matrix K nn still needs to be determined. Here, we follow the recommendation given in Ref. 40, where the noise of camera channel μ ¼ R; G; B under the v'th illuminant was estimated through the variance σ 2 μν in the signal value of some central region A surrounding the evaluation point pði; jÞ of which the spectral reflectance should be reconstructed. Assuming the noise of different camera channels to be independent, as stated in Sec. 2, its autocorrelation matrix can finally be expressed as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 7 ; 3 2 6 ; 5 9 8 K nn ¼ DðηÞ; where the 18 × 1 column vector E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 8 ; 3 2 6 ; 5 5 6 is formed by the single variances as described above calculated for each camera channel and LED illuminant.
It should be noted that in the present study, the digital camera is operated in raw image capturing mode. Instead of using a debayering algorithm, the camera response vector c ði;jÞ belonging to pði; jÞ is therefore obtained by averaging the corresponding signal values within the surrounding pixel region A. Furthermore, the spectral responsivities s R , s G , and s B of the Canon 750D, which are necessary to construct the Wiener Filter, were measured using a monochromator setup, as described in Ref. 41. Results are illustrated in Fig. 3. As can be seen, the Canon 750D used in our experiments is unresponsive to incident light below 400 nm and above 700 nm due to its built-in ultraviolet and infrared bandelimination filters. For this reason, we limit the range of all subsequent measurements and calculations to a spectral Fig. 3 Relative spectral responsivity of the Canon 750D color channels ðRGBÞ measured by using a monochromator setup. Note that the spectral responsivity of a camera system results from the combination of the spectral transmittance of the used objective, the spectral transmittance of the Bayer pattern, and the spectral sensitivity of the monochrome camera sensor underneath.  Journal of Electronic Imaging 053012-5 Sep∕Oct 2018 • Vol. 27 (5) range between these 400 and 700 nm with a 1 nm sampling, i.e., n ¼ 301.

Results of the Spectral Reflectance Estimation
In order to evaluate the accuracy of the Wiener estimation approach described in this paper, the results of both measured and estimated reflectances should be compared for a certain variety of organic tissue. Here, the term "measured" refers to the results obtained from direct radiometric measurements provided by a spectrometer, while the term "estimated" denotes the spectral reflectances that were reconstructed from RGB camera responses by applying the Wiener filter approach discussed above. The following tissue samples have been chosen: a pig's heart, tongue, stomach, lung, liver, spleen, and kidney as well as a freshly butchered beef steak representing muscle tissue. For the measurements of the spectral reflectances, the same measurement equipment and geometry, as shown in Fig. 1, was used with the radiance measuring head of the array spectrometer being placed on the top of the experimental box and connected to the fixture holding it in position. The measuring aperture was adjusted to 0.3 deg. The PWM values of the six LED channels were optimized to simulate CIE D65 illuminant and we measured the resulting SPD l w reflected by a calibrated white standard with known spectral reflectance r w , which had been positioned right beneath the spectrometer head, see Fig. 1(a). Afterward, the white standard was removed from the box and a tissue sample was placed at exactly the same position. We measured the light l sample being reflected from its surface. The spectral reflectance of the tissue sample can eventually be calculated by E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 9 ; 6 3 ; 4 1 0 r sample ¼ l sample · r w l w ; where the computation has to be performed wavelengthwise. Next, the measuring head was removed and replaced by the Canon 750D without touching or moving the organic tissue sample under inspection. Note that due to the specially designed fixture, the optical axis of the camera and of the spectrometer head could be adjusted to virtually lie exactly on top of each other so that, in both cases, the same evaluation point p 0 given by the intersection of the optical axis with the tissue surface could be chosen for direct measurement and reconstruction.
During image capturing, the six different LED channels were separately switch on and off one after the other and, for each of these illumination conditions, an image of the organic tissue sample was taken resulting in a total number of six RGB raw images per sample. The images were stored to disk and subsequently processed by applying the Wiener filter matrix W to the extracted camera response vector c at location p 0 , as described in Secs. 2 and 3, to obtain an estimater for the actual spectral reflectance r sample . To guarantee that the same spectral information reflected from the organic tissue surface was used for both the direct measurement and the estimation, the pixel area A surrounding p 0 , which is needed to calculate the variances of the noise matrix and the camera response vector (see Sec. 3), was limited to the size of the measuring spot of the spectrometer projected onto the surface of the tissue sample. This additional constraint allows for a better and more meaningful comparison between the direct measurement and the performance of the Wiener filter algorithm, since it eliminates in sufficiently good approximation, the uncertainties attributed to the different measuring geometries that otherwise would have had a large impact and, therefore, enables the proper determination of the scaling factor δ in the current experimental setup. Here, this factor was optimized in such a way that the spectral reflectance of a known color sample measured at the same position and under the same geometry as later the organic tissue samples was reconstructed as good as possible. With the same calibrated white standard as mentioned previously being chosen for this purpose, negligibly small deviations of less than 0.5% at each wavelength were observed between the known reflectance spectrum of the white standard and its Wiener filter estimate obtained for the optimal scaling factor δ opt: , which was subsequently used as a fixed value in the definition of the lighting-sensitivity matrix Λ given by Eq. (5) and, therefore, in the final Wiener filter matrix W to be applied to the camera response vector c. Please note that with the scaling factor δ being dependent on the measuring geometry, especially on the distance between tissue sample and illuminating light source, a thorough precalibration similar to the one described here must be provided by the manufacturers of such future MIS devices in order to allow for a proper in-situ reflectance estimation of organic tissue.
Finally, regarding the performance validation of the Wiener filter ansatz for such a purpose, the measurement protocol and experimental setup described in this and the preceding section, respectively, were used to reconstruct the spectral reflectances of several different organic tissue samples from digital camera responses. Figure 4 summarizes the obtained reconstruction results and compares them with the corresponding measured reflectances derived from Eq. (9). With the exception of the noisy part of the measured spectra below 430 nm, pretty good agreement can be observed for all samples. In Table 1, the degree of deviation between the estimated and measured reflectances is expressed in terms of the MSE metric and the CIEDE2000 color differences ΔE 00 . 42 Although the former is pretty common in statistics to be used for comparative purposes with a value of zero indicating perfect agreement, the latter being the current CIE recommendation for computing small color differences allows for drawing conclusions about the similarity of the visual appearance of the estimation and the measurement when hypothetically viewed under identical illumination conditions, which in the present case are assumed to be defined by reference illuminant D65. For completeness, the chromatic-only differences ΔE chrom: 00 are also tabulated.
As can be seen, relatively small MSE values ranging between 7.28 × 10 −4 and 2.61 × 10 −5 with an average of 2.15 × 10 −4 are obtained for various tissue samples. Compared to the MSE performance of the Wiener filter approach reported by other researchers, 22,38,43,44 where typical values were found to be of the order of ∼7 × 10 −4 , this basically indicates good to excellent absolute agreement between the estimated and measured reflectances. Furthermore, regarding the corresponding CIEDE2000 measure, most of the tissue samples, with the exception of lung and spleen, exhibit barely noticeable color differences ranging between 0.74 and 2.10 ΔE 00 with an average of 1.39 ΔE 00 close to the empirically determined just noticable limit defined by a ΔE 00 of unity. 45 Again, these results indicate excellent performance of the Wiener filter estimation and conform with previous findings. 44 Even the larger color differences obtained for the tissue samples of lung and spleen, which might be noticed also by unexperienced observers 45 (but only if they had the chance of comparing the reconstructed and the original tissue samples side by side), are not critical and still in an acceptable range because for the intended application of the spectral reconstruction algorithm in the present work, it is more important to reproduce the general trend of the in-situ reflectance spectra, as shown in Fig. 4, than to exactly match their theoretical visual appearance. In other words, even though the estimated tissue reflectances, which are used for optimizing the color correction matrix as described in the next section, show some kind of offset in comparison to their ground truth, the resulting linear color correction matrix-as we will see-still features excellent performance with respect to reproducing the color appearance of the original tissue samples.

Basic Method
With the in-situ spectral reflectances being known from spectral reconstruction, an optimized color correction becomes feasible. Following International Organization for Standardization (ISO) 17321-1:2012, 46 which defines the recommended standard method for the color characterization of digital cameras, a two-step optimization process is applied here. In the first step, an initial 3 × 3 color correction matrix M init is derived from a training set of k color patches, whose spectral reflectances r i are known, where i ¼ 1; : : : ; k.
Assuming D65 reference illuminant, the resulting camera responses c i for the i'th color patch can be calculated by using Eq. (2), while the corresponding tristimulus values t i ¼ ðt X;i ; t Y;i ; t Z;i Þ T are given by E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 0 ; 3 2 6 ; 7 0 8 is chosen in such a way that a perfectly diffusing, white color patch (spectral reflectance equals unity for all wavelengths) yields a tristimulus value t Y of 100.
With this input, the initial color correction matrix is eventually be obtained by solving the linear least-mean-squares optimization problem, which maps the camera responses onto the corresponding tristimulus values. It can be shown that the corresponding solution is given by Ref. 47: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 2 ; 3 2 6 ; 4 9 0 where T ¼ ðt 1 ; t 2 ; : : : ; t k Þ and C ¼ ðc 1 ; c 2 ; : : : ; c k Þ are 3 × k dimensional matrices concatenating the tristimulus and camera response values of the k color patches, respectively. As recommended by the ISO standard, the training set used to calculate M init in the present work consisted of 18 colored patches extracted from an X-Rite ColorChecker ® Classic and 21 neutral grey patches with their CIELAB correlate of lightness L Ã ranging from 12 to 92 in approximately equally spaced steps. Once the initial color correction matrix is found, one can proceed with the second step of optimization. According to the ISO standard, a nonlinear optimization technique should be applied in order to find the final color correction matrix, which minimizes the average squared perceived color difference between the color corrected and the actual tristimulus values of a second training set of color patches. For this purpose, the ISO standard recommends to use eight different color patches originally designated for calculating the so-called sensitivity metamerism index (SMI), which, using the framework of the CIE color rendition index, 48 gives a measure for the remaining colorimetric errors of digital camera systems. Hence, the resulting optimization problem can be formulated as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 3 ; 6 3 ; 1 9 2 M SMI ¼ argmin where the second constraint has been introduced to guarantee the white point preservation to reference illuminant D65 and L Ã a Ã b Ã ð: : : Þ denotes the transformation of the tristimulus values to CIELAB color space using the formulae given in Ref. 49. Again, the CIEDE2000 color difference formula ΔE 00 is applied here. By replacing the standard SMI colors in Eq. (13) with the estimated in-situ reflectance spectra of Fig. 4, a second color correction matrix M opt: can be calculated, which is now specifically optimized for accurately reproducing the color Journal of Electronic Imaging 053012-7 Sep∕Oct 2018 • Vol. 27 (5) properties of the organic tissue under inspection. In order to compare the performance of this more sophisticated color correction with the performance of the standard method, the eight original tissue samples of Fig. 4 have been chosen for testing with the corresponding results being reported in the following section.

Results of the Optimized Color Correction and
Further Implementation Notes Assuming D65 reference illuminant, the resulting CIEDE2000 color differences ΔE 00 between the color corrected and the actual tristimulus values of the eight original tissue samples selected for performance validation can be calculated using either of the two color correction matrices. The corresponding distributions of the obtained color differences are eventually compared and visualized in Fig. 5.
As can be seen, the application of the specifically optimized matrix M opt: for performing the color correction leads to heavily reduced color differences ΔE 00 for all eight tissue samples when compared to the application of the standard matrix M SMI . While the former shows an average color difference of only 0.42 ΔE 00 , the latter performs much worse giving an average color difference of 1.22 ΔE 00 , which is approximately three times larger. By making use of a two-sample t-test for unequal variances, 50,51 whose application requirements have been tested using the Shapiro-Wilk test [52][53][54] (checking for normality) in combination with the Brown-Forsythe extension of Levene's test 55 (checking for equal variances), the significance of the observed improvement for the optimized color correction matrix M opt: could clearly be confirmed (T ¼ 4.42, p ¼ 0.0016, α ¼ 0.05) showing a large effect size d 56 of 2.21. Hence, it can be concluded that especially in cases where a proper distinction of different kinds of tissue is necessary, significant enhancement in color correction by a factor of three might be crucial for the success of a surgery. Furthermore, with a demand for improved color reproduction properties of digital systems being reported for such kind of applications (see Sec. 1), these results clearly emphasize the importance of our in-situ approach for modern MIS surgery. In the attempt of providing a better visualization of the benefit of our proposed method, Fig. 6 exemplarily compares for the organic samples of lung and muscle tissue the respective color image representations of the debayered raw image data as being extracted from the Canon 750D imaging device before color correction and the color corrected image data obtained by applying M opt: to the raw camera responses at each pixel location. As can be seen, the image quality on the right-hand side of Fig. 6 associated with the optimized color correction seems to be much improved compared to what raw image processing would deliver: it is not just that the colors and tissue structures appear to be much more realistic, the color corrected images also provide more perceived depth and visual information than their uncorrected counterparts. Please note that in all cases, the same white balance was applied.
In practice, the whole process of recovering the true color information from captured RGB images can be implemented to work reliably and without any noticeable disruption of the surgical workflow simply by adopting a short flashing sequence of the different LED channels combined with a fast image capturing to create the necessary input data for running the Wiener filter estimation (see Sec. 3). Including the subsequent reconstruction of the targeted tissue reflectance spectra from the RGB camera responses, this can principally be realized to be accomplished within a few hundred milliseconds. With the application of the Wiener filter approach being basically given by a simple and fast-performing matrix multiplication [see Eq. (3)], the limiting factor in this context is the integration time of the actual imaging system. Assuming, for example, an MIS device which offers 60 fps, it would only take 100 ms for the flashing sequence and image capturing to be finished in case that the device light source consists of six different individual LED channels similar to the ones used in the present study. Hence, it can be concluded that the higher the frame rate the faster the flashing sequence and, therefore, the spectral reconstruction can be performed.
Regarding the issue of explicitly selecting the tissue samples, which are intended for being spectrally reconstructed, two complementary modes of operation are favored during a surgery. 7 First, the surgeon should be given the option to run an auto-tuned optimization of the color correction applied to the digital reproduction of the perceived surgical field. For this purpose, a certain number of arbitrary yet suitable pixel positions has to be selected automatically by the MIS device (the corresponding algorithm still needs to be developed) in order to perform the reflectance estimation of meaningful in-situ tissue samples that can eventually be used to enhance the color reproduction of the digital system following the methodology described above. Second, they should have the option to define the pixel positions on their own by using an appropriate, device-dependent input method. Besides allowing for an improved device color correction, which explicitly considers the spectral Fig. 5 Illustration of the CIEDE2000 color differences ΔE 00 between the color corrected and the actual tristimulus values of the original tissue samples shown in Fig. 4. The results of the standard color correction matrix M SMI are compared to those obtained for the specifically optimized matrix M opt: . As further emphasized by the corresponding box plots shown in the inset, the latter gives much smaller color differences for all tissue samples indicating superior color reproduction properties. With the statistical comparison of both methods showing a p value of 0.0016 and an effect size d of 2.21, a significant enhancement of the color correction for video-assisted surgery based on in-situ reflectance estimation could clearly be confirmed.
Journal of Electronic Imaging 053012-8 Sep∕Oct 2018 • Vol. 27 (5) characteristics of (critical) tissue samples judiciously selected by the surgeon, it also enables them to add further information and tissue metadata for archiving and teaching purposes. As a long-term goal, this kind of data in combination with the corresponding reconstructed reflectance spectra and, for example, an intelligent machine-learning algorithm could additionally be used (i) for automated tissue recognition as a support of the surgical workflow and (ii) to continuously extend and customize the database applied to train the Wiener filter leading to an improved performance of the MIS device for specific applications.
In any case, after having defined a certain number of device pixel positions (either automatically by the system or manually by the surgeon), the flashing sequence for enabling the reflectance estimation of the corresponding tissue samples must only be triggered once, i.e., with the exception of this quite short period of time, where the individual LED channels are switched on and off one after the other, the realtime image capturing is not affected by the applied Wiener filter estimation. The subsequent optimization of the color correction matrix can efficiently be performed on the microcontroller or CPU of the MIS device, which takes another few milliseconds and can easily be decoupled from the continuous image capturing so that like in the case of the Wiener filter estimation, this parallelly performed computation has no negative effect on the image quality, the contrast, or the actual imaging time of the MIS device. On the contrary, once the optimized color correction matrix has been calculated, it can be used to simply replace its standard counterpart in the imaging processing pipeline eventually leading to superior image quality and color reproduction properties as it has been shown previously in this work. However, it should again be noted here that the final implementation of the proposed method into a real video-assisted MIS device is still pending and, therefore, might entail further problems that one is not aware of yet.

Conclusion and Outlook
In this paper, an approach of optimizing the color reproduction properties of modern MIS devices based on the estimation of in-situ spectral reflectances applying the Wiener filter approach was proposed. Even though the Wiener filtering is widely used in image processing as a common way of reducing noise and image blurring 21,57-59 as well as in color science as a method for estimating camera sensitivities 39 and object reflectances from camera response data, [22][23][24][25][26]37,38,40 its usage described in the current paper is the first time that the results of such an estimation were adopted for the in-situ enhancement of the color reproduction properties of an imaging system based on the context-specific optimization of the corresponding color correction matrix for surgical applications. From this perspective, the current work should be considered as a first proof-of-concept, which, nonetheless, already clearly emphasizes the potential benefits resulting from the newly proposed method when being implemented into modern video-assisted MIS devices. In addition, it gives a detailed overview on how such an implementation can efficiently be performed and, at the same time, highlights further possible advantages such a system would offer like, for example, automated tissue recognition and true color archiving. Hence, it can be concluded that the present paper provides a significant contribution for future Fig. 6 Comparison between the exemplaric color image representations of the debayered raw image data (a and c) and the color corrected image data using M opt: (b and d) for the organic tissue samples of lung (upper row) and muscle (lower row). The blue square-shaped targets define the measurement points at various locations on the surface of the respective tissue sample. Please note that in all cases, the same white balance was applied.
Journal of Electronic Imaging 053012-9 Sep∕Oct 2018 • Vol. 27 (5) developments on the topic of video-assisted MIS surgery giving guidance for researchers and manufacturers working in this field on how to achieve superior image quality and additional system benefits.
Lacking an appropriate MIS device and the opportunity to test our approach under realistic conditions, an artificial abdomen/thorax was constructed to simulate the surgical workflow in the laboratory. While an array spectrometer connected to a radiance measuring head was used for the direct measurements of the spectral reflectances of test samples placed inside the thorax, the necessary input data for the Wiener filter to perform an accurate reflectance estimation were provided by a combination of a commercially available SLR digital camera and a customized six-channel LED light source. With this experimental setup, good to excellent agreement between the direct measurements and the reflectance estimation could be observed for a variety of different organic tissue samples.
Once the in-situ reflectance spectra were known from spectral reconstruction-at least to a sufficiently good approximation-it could be shown that an optimized color correction was feasible. Based on the results of the spectral reflectance estimation, a variation of the ISO standard method originally designated for the color characterization of digital cameras was applied here to derive a more sophisticated color correction matrix specifically optimized for accurately reproducing the color properties of the organic tissue samples under inspection. Statistical analysis eventually confirmed a significant enhancement in color reproduction when the proposed method is applied compared to what could be achieved by the standard procedure of colorimetric characterization. On average, three times smaller color differences could be observed, which, in a surgical application, may lead to less tissue damage and an increase in safety for the patient, especially in cases where a proper distinction of different kinds of tissue might be crucial for the success of a surgery.
Future research intentions mainly aim for porting, testing, and optimizing our approach on a real MIS device. In this context, it should be analyzed which LED channels are indispensable and which can be considered to be redundant in order to provide sufficiently good reconstruction results for achieving significantly enhanced color reproduction properties. So far, the LED selection in the current work was more or less arbitrary with the sole objective of being able to imitate reference illuminant D65 as good as possible. Even though verification is still pending, the blue LED channel in particular is supposed to be superfluous for the intended application of reconstructing the spectral reflectance properties of human tissue and could most likely be omitted in a prospective implementation.
Since the reported results so far were only obtained for ex vivo animal tissue samples, a further research question that still remained unanswered but should be addressed in the near future is whether the Wiener filter-based approach implemented for the current work is also suitable to be used for applications involving in vivo real human tissue. Due to biological similarity, it is indeed assumed that comparably excellent results could also be obtained in these cases-nevertheless, proof is still necessary and can only be provided after the proposed method has been successfully ported to a feature-complete MIS test device, which, as stated at the beginning of the last paragraph, should therefore be the main focus of further considerations.
e ¼ hðr −rÞ T ðr −rÞi (14) between the actual r and the estimatedr reflectance spectrum, where • denotes the expectation value operator. By applying Eq. (3), this MSE can be rewritten as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 5 ; 3 2 6 ; 6 0 3 e ¼ hr T r − r Tr −r T r þr Tr i ¼ hr T ri − hr Tr i − hr T ri þ hr Tr i ¼ hr T ri − hr T Wci − hc T W T ri þ hc T W T Wci; (15) where the camera response vector c is given by Eq.
(2) and consists of a combined lighting-sensitivity matrix Λ applied to the actual reflectance and an additive noise term n.
Mathematically, the task of finding a minimum of Eq. (15) with respect to W can be performed by setting its corresponding partial derivative to zero, i.e., E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 6 ; 3 2 6 ; 4 7 0 where the following matrix expressions were used: 60 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 7 ; 3 2 6 ; 4 1 8 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 8 ; 3 2 6 ; 3 7 6 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 9 ; 3 2 6 ; 3 3 9 ∂a T X T Xb ∂X ¼ Xðab T þ ba T Þ: Solving Eq. (16) for the Wiener filter matrix gives E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 0 ; 3 2 6 ; 2 9 0 Under the assumption of dealing with random noise of zero mean and variance σ, which is independent of the spectral reflectance, the subsequent equations hold for the respective expectation values: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 1 ; 3 2 6 ; 1 5 5 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 2 ; 3 2 6 ; 1 0 2 Journal of Electronic Imaging 053012-10 Sep∕Oct 2018 • Vol. 27 (5) Sebastian Babilon received his MSc degree in physics from the Technische Universität Darmstadt, Germany (2014). Since then, he has been working at the Laboratory of Lighting Technology in Darmstadt as a research assistant and doctoral candidate. His current work is focused on memory related color perception and the spectral optimization of multichannel LED luminaires for improved observer preference.
Paul Myland is a graduate student at the Laboratory of Lighting Technology at the Technische Universität Darmstadt, Germany.
He received his BSc degree in electrical engineering from the Technische Universität Darmstadt, Germany, in 2016. In his work, he was concerned with the application of a 3D-LUT approach for digital camera characterization. He was gathering industrial experience as an intern at ARRI Cine Technik in Munich, Germany. He is currently working on methods for a fast and reliable reconstruction of object reflectance spectra.
Julian Klabes is a graduate student at the Laboratory of Lighting Technology at the Technische Universität Darmstadt, Germany. He received his BSc degree in electrical engineering from the Technische Universität Darmstadt, Germany, in 2016. In his work, he was concerned with the application of various approaches for the characterization of digital display devices. His current work is focused on algorithms for object recognition based on surface reflectance spectra.
Joel Simon is an undergraduate student at the Laboratory of Lighting Technology at the Technische Universität Darmstadt, Germany. As part of a student research project, he developed, built, and characterized the experimental device described in this work, which was used for spectral reflectance measurements and reconstruction.
Tran Quoc Khanh is a university professor and head of the Laboratory of Lighting Technology at the Technische Universität Darmstadt, Germany. He graduated in optical technologies and obtained a doctoral degree in lighting engineering from the Technische Universität Ilmenau, Germany. Before being appointed as a professor, he gathered industrial experience as a project manager at Arnold and Richter Cine Technik AG in Munich, Germany.
His research interests cover all important aspects of modern lighting technology, including LEDs, colorimetry, mesopic vision, glare, photometry, and color science related topics.