*in-situ*spectral reflectances is proposed. Following the International Organization for Standardization (ISO) standard procedure of colorimetric characterization, an initial color correction matrix was determined first by solving a linear least-mean-squares optimization problem for a small set of artificial color samples mapping the corresponding camera responses onto the samples’ tristimulus values. This initial matrix was then used as a starting point for a second, nonlinear optimization, which makes use of the estimated reflectance spectra in order to minimize the average squared color difference between the color corrected and the actually perceived tristimulus values of the individual organic tissue samples. Compared to the ISO standard, which only knows simple color patches to be used for the nonlinear optimization, a significant enhancement in the color reproduction of organic tissue could be confirmed with the newly proposed method being applied instead.

## 1.

## Introduction

Over the past few decades, minimally invasive surgery (MIS) techniques have become prevalent in various fields of surgical diagnostics and therapy,^{1} offering many benefits over traditional surgery. These benefits include less tissue damage and, therefore, less pain for the patients, shorter recovery time as well as increased safety and precision.^{2}^{–}^{4} Moreover, the use of MIS procedures instead of traditional open surgery, especially for routine operations, can lead to significantly reduced health care costs.^{5}^{,}^{6} Researchers at Johns Hopkins University School of Medicine recently found that by increasing the use of laparoscopic techniques by 50%, US-American hospitals collectively could prevent thousands of postsurgical complications and save up to $280 million per year.^{6}

Currently, the majority of modern MIS systems are video based, giving surgeons an extended, in some cases, even three-dimensional view inside the patient’s body revealing its anatomical topography and, therefore, allowing for better visual support during surgery. Making use of digital endoscopes, such systems enable surgeons to operate through a much smaller incision than would otherwise be required for traditional open surgery. At the same time, the surgeon’s exact view of the surgical field can be provided to nurses or assisting surgeons with the same image quality by projecting the digital image of the MIS system on an additional monitor. This allows for better communication of surgical workflows and strategies and, therefore, more efficient collaboration in the operating room.

Despite the advantages of video-based MIS technologies and their increasing dissemination, there is still a lack of international standards and norms regarding the color reproduction properties of such fully digital visual systems. Modern endoscopic devices usually consist of a digital RGB camera system attached to a thin rigid or flexible tube, which can be inserted through small incisions or natural orifices (e.g., mouth or nostrils) into the patient’s body. The endoscopic tube is designed in such a way that tiny surgical instruments can be inserted and used by the surgeon in order to reach out for and operate on the internal organs. It further contains a carefully optimized optical fiber and lens system. In combination with an external light source and the attached camera system, this ensures both the proper illumination of the organs under inspection and the correct transmission of the image from the objective lens to the capturing device, i.e., the digital camera.

However, image quality and color reproduction still remain a big issue for such medical applications, which—according to the authors’ best knowledge—is addressed by only a few manufacturers because of missing obligatory standards in this field. Regarding other fields of application of digital camera systems, it is a known consensus that the issue of color reproduction has to be considered thoroughly to obtain qualitatively good and pleasant results. In this context, an interview-based investigation of the color image preference of surgeons revealed that a high acceptance of fully digital MIS devices could only be achieved if the experienced visual impressions resemble those usually perceived when using purely optical systems, i.e., systems they got used to over the years, which basically provide genuine colorimetric information.^{7} Hence, even though a good surgeon under the assumption of sufficient image contrast might still efficiently perform a surgery, in case various organic tissues were represented in colors deviating from their actual ground truth, there is strong evidence that a proper color reproduction is desirable to achieve high user acceptance. Furthermore, it is the authors’ belief that a colorimetrically genuine representation of the captured image data also provides an indispensable part for appropriate digital archiving and teaching purposes.

Assuming raw image capturing, the color reproduction of digital endoscopes and MIS devices is mainly influenced by the camera color filters, the optical transmission properties of the endoscopic tube, and the spectrum of the illuminating light source, which in modern solutions is formed by a highly efficient, multispectral LED light engine usually tuned to the emission of cool white light. In this context, perfect color reproduction means that the reproduced colors of, for example, inner organs would exactly match their direct perception under the same illumination. With this in mind, one can easily imagine that a more or less perfect color reproduction and, therefore, negligibly small colorimetric errors are crucial for the success of a surgery, where different kinds of tissues must be distinguished by the surgeon not only through their texture but also through their color. However, perfect color reproduction is usually not the case because, in general, the RGB-sensor of a digital camera system does not fulfill the required Luther–Ives condition,^{8}^{–}^{10} i.e., the spectral response curves of the RGB-sensor cannot be described as a linear combination of the eye cone response functions. As a consequence, more or less severe colorimetric errors occur when comparing the direct perception of the surgical field to its reproduction.

To correct for these systematic errors of digital cameras, various linear and nonlinear characterization approaches and methods have been proposed in the literature.^{11}^{–}^{20} Despite their varying degree of complexity, they all have in common that, for a given illuminant, they try to minimize the perceived color differences between the reproduction of some test colors and the direct perception of the same test colors in a device-independant uniform color space (UCS). In practice, most manufacturers apply a simple linear least-mean-squares optimization based on some standardized color charts, as described by Hubel et al.^{20} However, as one might expect, the reflectance properties of such printed color charts generally have nothing to do with those of real human tissue a surgeon is confronted with in the operating room and, therefore, such an optimization can only be considered as a rough estimate rather than a colorimetrically proper characterization. A significant improvement could only be achieved if the *in-situ* reflectance properties of the human tissue under inspection were known, which is usually not the case. The aim of this paper, therefore, is to provide an easy-to-implement reconstruction approach based on the Wiener filter^{21} that estimates the spectral reflectances of organic tissue from a series of images captured under varying LED illumination. It should be further shown that once the *in-situ* spectral reflectances are known, significantly improved color correction is feasible.

The paper is organized as follows. In Sec. 2, the applied Wiener-filter approach for estimating the spectral reflectances is introduced. Section 3 gives an overview of the experimental setup, whereas in Sec. 4, some results of the spectral reflectance estimation are presented. In Sec. 5, a method based on *in-situ* measured reflectance spectra is discussed, which attempts to enhance the color correction properties of video-assisted MIS devices. Finally, the paper closes in Sec. 6 with some concluding remarks and an outlook on future research intentions.

## 2.

## Wiener-Filter Estimation Approach

Estimating *in-situ* spectral reflectances from digital camera responses of a modern endoscope generally is an ill-posed problem^{22}^{,}^{23} since the dimensionality of the camera signals is much smaller than the number of spectral components that should be reconstructed. In the literature, several strategies have been proposed to tackle this problem making use of *a priori* knowledge of the underlying acquisition process.^{12}^{,}^{22}^{–}^{36} In this context, one of the most important and widely used reconstruction methods is the so-called Wiener estimation, which is also the method of choice in the present work. Despite its simplicity, the Wiener estimation approach has been proven to provide accurate results regarding the spectral reconstruction of nonflourescent natural reflectances.^{22}^{–}^{26}

Assuming a normal distribution and statistic independence of reflectances and system noise, the reflectance-estimating Wiener Filter, which is applied to the camera responses, minimizes the mean-squared error (MSE) between the estimated and the actual reflectance of a surface color. Basically, the responses ${\mathbf{c}}^{(i,j)}={[{c}_{R}^{(i,j)},{c}_{G}^{(i,j)},{c}_{B}^{(i,j)}]}^{T}$ of a linear trichromatic RGB camera system at a pixel location $(i,j)$ on the camera sensor are given by

## Eq. (1)

$${\mathbf{c}}^{(i,j)}={\delta}^{(i,j)}\mathbf{S}D(\mathbf{l}){\mathbf{r}}^{(i,j)}+{\mathbf{n}}^{(i,j)},$$For the sake of simplicity, the pixel position indices $(i,j)$ are omitted in the following and Eq. (1) can be rewritten as follows:

where $\mathbf{\Lambda}=\delta \mathbf{S}D(\mathbf{l})$ is the combined lighting-sensitivity matrix of the system. In order to solve the above equation and to provide an accurate estimate $\widehat{\mathbf{r}}$ for the actual reflectance, a Wiener filter matrix $\mathbf{W}$ is introduced to be applied to the RGB camera responses, which readsAccording to Refs. 2324.–25 and 37, the above reflectance-estimating Wiener filter matrix is defined in such a way that it minimizes the MSE between the estimated and the actual reflectance and, therefore, can be derived as follows:

## Eq. (4)

$$\mathbf{W}={\mathbf{K}}_{\mathbf{rr}}{\mathbf{\Lambda}}^{T}{(\mathbf{\Lambda}{\mathbf{K}}_{\mathbf{rr}}{\mathbf{\Lambda}}^{T}+{\mathbf{K}}_{\mathbf{nn}})}^{-1},$$## 3.

## Experimental Setup

Lacking a real MIS device equipped with a multispectral LED light source, we decided to set up an experimental environment, as shown in Figs. 1(a) and 1(b), which simulates the surgical workflow using a commercially available single lens reflex (SLR) camera and a customized six-channel LED light engine. Here, the abdomen or thorax of a patient is abstracted by a 40 cm × 40 cm × 60 cm wooden box with diffusely reflecting white painted walls on the inside in order to guarantee homogeneous illumination of the organic tissue under inspection. Please keep in mind that this setup should only provide a first proof-of-concept rather than mirroring reality.

As can be seen from Fig. 1(a), the illumination is realized from above by a six-channel (red, deep red, green, blue, warm-white, and cool-white) edge-LED arrangement consisting of multicolor LED stripes being installed on all four sides of the box’s lid. Several diffusing panels covering the LEDs and forming the bottom of the lid guarantee diffuse and homogeneous light emission into the artificial abdomen/thorax. An aperture ring with a diameter of 10 cm embedded in the center of the lid together with a specially designed fixture allows for an easy insertion and alignment of different measurement devices. Furthermore, each LED channel can be controlled separately via pulse-width modulated (PWM) dimming and the corresponding normalized SPDs are illustrated in Fig. 2. They were measured in $0\text{\hspace{0.17em}}\mathrm{deg}/0\text{\hspace{0.17em}}\mathrm{deg}$ geometry using a white reflectance standard placed centrally inside the experimental box in combination with an Instrument Systems CAS 140CT array spectrometer connected to a TOP 200 radiance measuring head, as shown in Fig. 1(b).

During the actual experiment, i.e., the image capturing and subsequent spectral reconstruction of organic tissue reflectances, the spectrometer head is removed from the box and a Canon 750D SLR digital camera is inserted at exactly the same position with its objective facing perpendicularly down toward the test object being placed right beneath the camera in the center of the bottom of the experimental box. Instead of human organs, freshly butchered pig and cattle organs were used as test objects to investigate the performance of the reconstruction algorithm. Due to biological similarity, it is supposed that results can easily be transferred to MIS applications involving human tissue.

As mentioned in Sec. 2, an easy-to-implement Wiener filter approach is used to estimate the spectral reflectances of the different test organs from the digital $RGB$ camera responses of the Canon 750D. One problem that arises with this estimation method is that more than three camera responses are usually needed to obtain high accuracy.^{38} This is because of the extremely ill-posed nature of the calculation when mapping only three values $R$, $G$, and $B$ to reflectance intensities at a few hundred different wavelengths (e.g., 401 different wavelengths from 380 to 780 nm if 1 nm sampling is applied). In order to overcome this problem, we combine multiple captures of the test organs illuminated by different LED SPDs [six in our case, see Fig. 2]. As stated in Ref. 39, this allows us to significantly increase the effective dimension and, as a result, make a more accurate estimate of the spectral reflectances without the need of additional constraining assumptions on their properties (positivity, boundedness, and smoothness must be fulfilled for naturally occurring reflectance spectra).

Hence, the reflectance-estimating Wiener filter of Eq. (4) needs to be adjusted accordingly. Considering the six different illumination conditions ${\mathbf{l}}_{1},\dots ,{\mathbf{l}}_{6}$ of our experimental box, the combined lighting-sensitivity matrix $\mathbf{\Lambda}$ must be rewritten as follows:

## Eq. (5)

$$\mathbf{\Lambda}=\delta {[D({\mathbf{l}}_{1}){\mathbf{S}}^{T},\dots ,D({\mathbf{l}}_{6}){\mathbf{S}}^{T}]}^{T},$$Last but not least, the noise autocorrelation matrix ${\mathbf{K}}_{\mathbf{nn}}$ still needs to be determined. Here, we follow the recommendation given in Ref. 40, where the noise of camera channel $\mu =R,G,B$ under the $v$’th illuminant was estimated through the variance ${\sigma}_{\mu \nu}^{2}$ in the signal value of some central region $A$ surrounding the evaluation point $p(i,j)$ of which the spectral reflectance should be reconstructed. Assuming the noise of different camera channels to be independent, as stated in Sec. 2, its autocorrelation matrix can finally be expressed as follows:

where the $18\times 1$ column vector## Eq. (8)

$$\mathit{\eta}={({\sigma}_{R1}^{2},{\sigma}_{G1}^{2},{\sigma}_{B1}^{2},\dots ,{\sigma}_{R6}^{2},{\sigma}_{G6}^{2},{\sigma}_{B6}^{2})}^{T}$$It should be noted that in the present study, the digital camera is operated in raw image capturing mode. Instead of using a debayering algorithm, the camera response vector ${\mathbf{c}}^{(i,j)}$ belonging to $p(i,j)$ is therefore obtained by averaging the corresponding signal values within the surrounding pixel region $A$. Furthermore, the spectral responsivities ${\mathbf{s}}_{R}$, ${\mathbf{s}}_{G}$, and ${\mathbf{s}}_{B}$ of the Canon 750D, which are necessary to construct the Wiener Filter, were measured using a monochromator setup, as described in Ref. 41. Results are illustrated in Fig. 3. As can be seen, the Canon 750D used in our experiments is unresponsive to incident light below 400 nm and above 700 nm due to its built-in ultraviolet and infrared band-elimination filters. For this reason, we limit the range of all subsequent measurements and calculations to a spectral range between these 400 and 700 nm with a 1 nm sampling, i.e., $n=301$.

## 4.

## Results of the Spectral Reflectance Estimation

In order to evaluate the accuracy of the Wiener estimation approach described in this paper, the results of both measured and estimated reflectances should be compared for a certain variety of organic tissue. Here, the term “measured” refers to the results obtained from direct radiometric measurements provided by a spectrometer, while the term “estimated” denotes the spectral reflectances that were reconstructed from $RGB$ camera responses by applying the Wiener filter approach discussed above. The following tissue samples have been chosen: a pig’s heart, tongue, stomach, lung, liver, spleen, and kidney as well as a freshly butchered beef steak representing muscle tissue.

For the measurements of the spectral reflectances, the same measurement equipment and geometry, as shown in Fig. 1, was used with the radiance measuring head of the array spectrometer being placed on the top of the experimental box and connected to the fixture holding it in position. The measuring aperture was adjusted to 0.3 deg. The PWM values of the six LED channels were optimized to simulate CIE D65 illuminant and we measured the resulting SPD ${\mathbf{l}}_{\mathrm{w}}$ reflected by a calibrated white standard with known spectral reflectance ${\mathbf{r}}_{\mathrm{w}}$, which had been positioned right beneath the spectrometer head, see Fig. 1(a). Afterward, the white standard was removed from the box and a tissue sample was placed at exactly the same position. We measured the light ${\mathbf{l}}_{\text{sample}}$ being reflected from its surface. The spectral reflectance of the tissue sample can eventually be calculated by

## Eq. (9)

$${\mathbf{r}}_{\text{sample}}=\frac{{\mathbf{l}}_{\text{sample}}\xb7{\mathbf{r}}_{\mathrm{w}}}{{\mathbf{l}}_{\mathrm{w}}},$$Next, the measuring head was removed and replaced by the Canon 750D without touching or moving the organic tissue sample under inspection. Note that due to the specially designed fixture, the optical axis of the camera and of the spectrometer head could be adjusted to virtually lie exactly on top of each other so that, in both cases, the same evaluation point ${p}_{0}$ given by the intersection of the optical axis with the tissue surface could be chosen for direct measurement and reconstruction.

During image capturing, the six different LED channels were separately switch on and off one after the other and, for each of these illumination conditions, an image of the organic tissue sample was taken resulting in a total number of six $RGB$ raw images per sample. The images were stored to disk and subsequently processed by applying the Wiener filter matrix $\mathbf{W}$ to the extracted camera response vector $\mathbf{c}$ at location ${p}_{0}$, as described in Secs. 2 and 3, to obtain an estimate $\widehat{\mathbf{r}}$ for the actual spectral reflectance ${\mathbf{r}}_{\text{sample}}$. To guarantee that the same spectral information reflected from the organic tissue surface was used for both the direct measurement and the estimation, the pixel area $A$ surrounding ${p}_{0}$, which is needed to calculate the variances of the noise matrix and the camera response vector (see Sec. 3), was limited to the size of the measuring spot of the spectrometer projected onto the surface of the tissue sample. This additional constraint allows for a better and more meaningful comparison between the direct measurement and the performance of the Wiener filter algorithm, since it eliminates in sufficiently good approximation, the uncertainties attributed to the different measuring geometries that otherwise would have had a large impact and, therefore, enables the proper determination of the scaling factor $\delta $ in the current experimental setup. Here, this factor was optimized in such a way that the spectral reflectance of a known color sample measured at the same position and under the same geometry as later the organic tissue samples was reconstructed as good as possible. With the same calibrated white standard as mentioned previously being chosen for this purpose, negligibly small deviations of less than 0.5% at each wavelength were observed between the known reflectance spectrum of the white standard and its Wiener filter estimate obtained for the optimal scaling factor ${\delta}_{\mathrm{opt.}}$, which was subsequently used as a fixed value in the definition of the lighting-sensitivity matrix $\mathbf{\Lambda}$ given by Eq. (5) and, therefore, in the final Wiener filter matrix $\mathbf{W}$ to be applied to the camera response vector $\mathbf{c}$. Please note that with the scaling factor $\delta $ being dependent on the measuring geometry, especially on the distance between tissue sample and illuminating light source, a thorough precalibration similar to the one described here must be provided by the manufacturers of such future MIS devices in order to allow for a proper *in-situ* reflectance estimation of organic tissue.

Finally, regarding the performance validation of the Wiener filter ansatz for such a purpose, the measurement protocol and experimental setup described in this and the preceding section, respectively, were used to reconstruct the spectral reflectances of several different organic tissue samples from digital camera responses. Figure 4 summarizes the obtained reconstruction results and compares them with the corresponding measured reflectances derived from Eq. (9). With the exception of the noisy part of the measured spectra below 430 nm, pretty good agreement can be observed for all samples. In Table 1, the degree of deviation between the estimated and measured reflectances is expressed in terms of the MSE metric and the CIEDE2000 color differences $\mathrm{\Delta}{E}_{00}$.^{42} Although the former is pretty common in statistics to be used for comparative purposes with a value of zero indicating perfect agreement, the latter being the current CIE recommendation for computing small color differences allows for drawing conclusions about the similarity of the visual appearance of the estimation and the measurement when hypothetically viewed under identical illumination conditions, which in the present case are assumed to be defined by reference illuminant D65. For completeness, the chromatic-only differences $\mathrm{\Delta}{E}_{00}^{\mathrm{chrom.}}$ are also tabulated.

## Table 1

Overview of the observed degree of deviation between the estimated and measured reflectance spectra of various organic tissue samples in terms of MSE, ΔE00, and ΔE00chrom. values. Due to the large measurement noise below 430 nm, only signals at larger wavelengths were considered for the calculations. In addition, reference illuminant D65 was assumed for computing the color differences.

Steak | Heart | Liver | Kidney | Lung | Stomach | Tongue | Spleen | |
---|---|---|---|---|---|---|---|---|

$\mathrm{\Delta}{E}_{00}$ | 2.10 | 0.74 | 1.63 | 1.50 | 4.08 | 1.32 | 1.07 | 3.66 |

$\mathrm{\Delta}{E}_{00}^{\mathrm{chrom.}}$ | 2.04 | 0.37 | 1.35 | 1.49 | 2.51 | 0.42 | 0.99 | 3.60 |

$\mathrm{MSE}\times {10}^{4}$ | 0.50 | 0.97 | 0.35 | 0.26 | 6.32 | 7.28 | 0.71 | 0.79 |

As can be seen, relatively small MSE values ranging between $7.28\times {10}^{-4}$ and $2.61\times {10}^{-5}$ with an average of $2.15\times {10}^{-4}$ are obtained for various tissue samples. Compared to the MSE performance of the Wiener filter approach reported by other researchers,^{22}^{,}^{38}^{,}^{43}^{,}^{44} where typical values were found to be of the order of $\sim 7\times {10}^{-4}$, this basically indicates good to excellent absolute agreement between the estimated and measured reflectances. Furthermore, regarding the corresponding CIEDE2000 measure, most of the tissue samples, with the exception of lung and spleen, exhibit barely noticeable color differences ranging between 0.74 and $2.10\text{\hspace{0.17em}}\mathrm{\Delta}{E}_{00}$ with an average of $1.39\text{\hspace{0.17em}}\mathrm{\Delta}{E}_{00}$ close to the empirically determined just noticable limit defined by a $\mathrm{\Delta}{E}_{00}$ of unity.^{45} Again, these results indicate excellent performance of the Wiener filter estimation and conform with previous findings.^{44} Even the larger color differences obtained for the tissue samples of lung and spleen, which might be noticed also by unexperienced observers^{45} (but only if they had the chance of comparing the reconstructed and the original tissue samples side by side), are not critical and still in an acceptable range because for the intended application of the spectral reconstruction algorithm in the present work, it is more important to reproduce the general trend of the *in-situ* reflectance spectra, as shown in Fig. 4, than to exactly match their theoretical visual appearance. In other words, even though the estimated tissue reflectances, which are used for optimizing the color correction matrix as described in the next section, show some kind of offset in comparison to their ground truth, the resulting linear color correction matrix—as we will see—still features excellent performance with respect to reproducing the color appearance of the original tissue samples.

## 5.

## Enhanced Color Correction for Video-Assisted Surgery

## 5.1.

### Basic Method

With the *in-situ* spectral reflectances being known from spectral reconstruction, an optimized color correction becomes feasible. Following International Organization for Standardization (ISO) 17321-1:2012,^{46} which defines the recommended standard method for the color characterization of digital cameras, a two-step optimization process is applied here. In the first step, an initial $3\times 3$ color correction matrix ${\mathbf{M}}_{\mathrm{init}}$ is derived from a training set of $k$ color patches, whose spectral reflectances ${\mathbf{r}}_{i}$ are known, where $i=1,\dots ,k$. Assuming D65 reference illuminant, the resulting camera responses ${\mathbf{c}}_{i}$ for the $i$’th color patch can be calculated by using Eq. (2), while the corresponding tristimulus values ${\mathbf{t}}_{i}={({t}_{X,i},{t}_{Y,i},{t}_{Z,i})}^{T}$ are given by

With this input, the initial color correction matrix is eventually be obtained by solving the linear least-mean-squares optimization problem, which maps the camera responses onto the corresponding tristimulus values. It can be shown that the corresponding solution is given by Ref. 47:

## Eq. (12)

$${\mathbf{M}}_{\mathrm{init}}={\mathbf{TC}}^{\mathbf{T}}{({\mathbf{CC}}^{\mathbf{T}})}^{-\mathbf{1}},$$^{®}Classic and 21 neutral grey patches with their CIELAB correlate of lightness ${L}^{*}$ ranging from 12 to 92 in approximately equally spaced steps.

Once the initial color correction matrix is found, one can proceed with the second step of optimization. According to the ISO standard, a nonlinear optimization technique should be applied in order to find the final color correction matrix, which minimizes the average squared perceived color difference between the color corrected and the actual tristimulus values of a second training set of color patches. For this purpose, the ISO standard recommends to use eight different color patches originally designated for calculating the so-called sensitivity metamerism index (SMI), which, using the framework of the CIE color rendition index,^{48} gives a measure for the remaining colorimetric errors of digital camera systems. Hence, the resulting optimization problem can be formulated as follows:

## Eq. (13)

$${\mathbf{M}}_{\mathrm{SMI}}=\underset{{\mathbf{M}}_{\mathrm{SMI}}}{\mathrm{argmin}}\left(\frac{1}{8}\sum _{i=1}^{8}{\{\mathrm{\Delta}{E}_{00}[{L}^{*}{a}^{*}{b}^{*}({\mathbf{M}}_{\mathrm{SMI}}{\mathbf{c}}_{\mathrm{SMI},\mathrm{i}}),{L}^{*}{a}^{*}{b}^{*}({\mathbf{t}}_{\mathrm{SMI},i})]\}}^{2}\right),$$By replacing the standard SMI colors in Eq. (13) with the estimated *in-situ* reflectance spectra of Fig. 4, a second color correction matrix ${\mathbf{M}}_{\mathrm{opt}.}$ can be calculated, which is now specifically optimized for accurately reproducing the color properties of the organic tissue under inspection. In order to compare the performance of this more sophisticated color correction with the performance of the standard method, the eight original tissue samples of Fig. 4 have been chosen for testing with the corresponding results being reported in the following section.

## 5.2.

### Results of the Optimized Color Correction and Further Implementation Notes

Assuming D65 reference illuminant, the resulting CIEDE2000 color differences $\mathrm{\Delta}{E}_{00}$ between the color corrected and the actual tristimulus values of the eight original tissue samples selected for performance validation can be calculated using either of the two color correction matrices. The corresponding distributions of the obtained color differences are eventually compared and visualized in Fig. 5.

As can be seen, the application of the specifically optimized matrix ${\mathbf{M}}_{\mathrm{opt.}}$ for performing the color correction leads to heavily reduced color differences $\mathrm{\Delta}{E}_{00}$ for all eight tissue samples when compared to the application of the standard matrix ${\mathbf{M}}_{\mathrm{SMI}}$. While the former shows an average color difference of only $0.42\text{\hspace{0.17em}}\mathrm{\Delta}{E}_{00}$, the latter performs much worse giving an average color difference of $1.22\text{\hspace{0.17em}}\mathrm{\Delta}{E}_{00}$, which is approximately three times larger. By making use of a two-sample $t$-test for unequal variances,^{50}^{,}^{51} whose application requirements have been tested using the Shapiro–Wilk test^{52}^{–}^{54} (checking for normality) in combination with the Brown–Forsythe extension of Levene’s test^{55} (checking for equal variances), the significance of the observed improvement for the optimized color correction matrix ${\mathbf{M}}_{\mathrm{opt.}}$ could clearly be confirmed ($T=4.42$, $p=0.0016$, $\alpha =0.05$) showing a large effect size $d$^{56} of 2.21. Hence, it can be concluded that especially in cases where a proper distinction of different kinds of tissue is necessary, significant enhancement in color correction by a factor of three might be crucial for the success of a surgery. Furthermore, with a demand for improved color reproduction properties of digital systems being reported for such kind of applications (see Sec. 1), these results clearly emphasize the importance of our *in-situ* approach for modern MIS surgery. In the attempt of providing a better visualization of the benefit of our proposed method, Fig. 6 exemplarily compares for the organic samples of lung and muscle tissue the respective color image representations of the debayered raw image data as being extracted from the Canon 750D imaging device before color correction and the color corrected image data obtained by applying ${\mathbf{M}}_{\mathrm{opt.}}$ to the raw camera responses at each pixel location. As can be seen, the image quality on the right-hand side of Fig. 6 associated with the optimized color correction seems to be much improved compared to what raw image processing would deliver: it is not just that the colors and tissue structures appear to be much more realistic, the color corrected images also provide more perceived depth and visual information than their uncorrected counterparts. Please note that in all cases, the same white balance was applied.

In practice, the whole process of recovering the true color information from captured $RGB$ images can be implemented to work reliably and without any noticeable disruption of the surgical workflow simply by adopting a short flashing sequence of the different LED channels combined with a fast image capturing to create the necessary input data for running the Wiener filter estimation (see Sec. 3). Including the subsequent reconstruction of the targeted tissue reflectance spectra from the $RGB$ camera responses, this can principally be realized to be accomplished within a few hundred milliseconds. With the application of the Wiener filter approach being basically given by a simple and fast-performing matrix multiplication [see Eq. (3)], the limiting factor in this context is the integration time of the actual imaging system. Assuming, for example, an MIS device which offers 60 fps, it would only take 100 ms for the flashing sequence and image capturing to be finished in case that the device light source consists of six different individual LED channels similar to the ones used in the present study. Hence, it can be concluded that the higher the frame rate the faster the flashing sequence and, therefore, the spectral reconstruction can be performed.

Regarding the issue of explicitly selecting the tissue samples, which are intended for being spectrally reconstructed, two complementary modes of operation are favored during a surgery.^{7} First, the surgeon should be given the option to run an auto-tuned optimization of the color correction applied to the digital reproduction of the perceived surgical field. For this purpose, a certain number of arbitrary yet suitable pixel positions has to be selected automatically by the MIS device (the corresponding algorithm still needs to be developed) in order to perform the reflectance estimation of meaningful *in-situ* tissue samples that can eventually be used to enhance the color reproduction of the digital system following the methodology described above. Second, they should have the option to define the pixel positions on their own by using an appropriate, device-dependent input method. Besides allowing for an improved device color correction, which explicitly considers the spectral characteristics of (critical) tissue samples judiciously selected by the surgeon, it also enables them to add further information and tissue metadata for archiving and teaching purposes. As a long-term goal, this kind of data in combination with the corresponding reconstructed reflectance spectra and, for example, an intelligent machine-learning algorithm could additionally be used (i) for automated tissue recognition as a support of the surgical workflow and (ii) to continuously extend and customize the database applied to train the Wiener filter leading to an improved performance of the MIS device for specific applications.

In any case, after having defined a certain number of device pixel positions (either automatically by the system or manually by the surgeon), the flashing sequence for enabling the reflectance estimation of the corresponding tissue samples must only be triggered once, i.e., with the exception of this quite short period of time, where the individual LED channels are switched on and off one after the other, the real-time image capturing is not affected by the applied Wiener filter estimation. The subsequent optimization of the color correction matrix can efficiently be performed on the microcontroller or CPU of the MIS device, which takes another few milliseconds and can easily be decoupled from the continuous image capturing so that like in the case of the Wiener filter estimation, this parallelly performed computation has no negative effect on the image quality, the contrast, or the actual imaging time of the MIS device. On the contrary, once the optimized color correction matrix has been calculated, it can be used to simply replace its standard counterpart in the imaging processing pipeline eventually leading to superior image quality and color reproduction properties as it has been shown previously in this work. However, it should again be noted here that the final implementation of the proposed method into a real video-assisted MIS device is still pending and, therefore, might entail further problems that one is not aware of yet.

## 6.

## Conclusion and Outlook

In this paper, an approach of optimizing the color reproduction properties of modern MIS devices based on the estimation of *in-situ* spectral reflectances applying the Wiener filter approach was proposed. Even though the Wiener filtering is widely used in image processing as a common way of reducing noise and image blurring^{21}^{,}^{57}^{–}^{59} as well as in color science as a method for estimating camera sensitivities^{39} and object reflectances from camera response data,^{22}^{–}^{26}^{,}^{37}^{,}^{38}^{,}^{40} its usage described in the current paper is the first time that the results of such an estimation were adopted for the *in-situ* enhancement of the color reproduction properties of an imaging system based on the context-specific optimization of the corresponding color correction matrix for surgical applications. From this perspective, the current work should be considered as a first proof-of-concept, which, nonetheless, already clearly emphasizes the potential benefits resulting from the newly proposed method when being implemented into modern video-assisted MIS devices. In addition, it gives a detailed overview on how such an implementation can efficiently be performed and, at the same time, highlights further possible advantages such a system would offer like, for example, automated tissue recognition and true color archiving. Hence, it can be concluded that the present paper provides a significant contribution for future developments on the topic of video-assisted MIS surgery giving guidance for researchers and manufacturers working in this field on how to achieve superior image quality and additional system benefits.

Lacking an appropriate MIS device and the opportunity to test our approach under realistic conditions, an artificial abdomen/thorax was constructed to simulate the surgical workflow in the laboratory. While an array spectrometer connected to a radiance measuring head was used for the direct measurements of the spectral reflectances of test samples placed inside the thorax, the necessary input data for the Wiener filter to perform an accurate reflectance estimation were provided by a combination of a commercially available SLR digital camera and a customized six-channel LED light source. With this experimental setup, good to excellent agreement between the direct measurements and the reflectance estimation could be observed for a variety of different organic tissue samples.

Once the *in-situ* reflectance spectra were known from spectral reconstruction—at least to a sufficiently good approximation—it could be shown that an optimized color correction was feasible. Based on the results of the spectral reflectance estimation, a variation of the ISO standard method originally designated for the color characterization of digital cameras was applied here to derive a more sophisticated color correction matrix specifically optimized for accurately reproducing the color properties of the organic tissue samples under inspection. Statistical analysis eventually confirmed a significant enhancement in color reproduction when the proposed method is applied compared to what could be achieved by the standard procedure of colorimetric characterization. On average, three times smaller color differences could be observed, which, in a surgical application, may lead to less tissue damage and an increase in safety for the patient, especially in cases where a proper distinction of different kinds of tissue might be crucial for the success of a surgery.

Future research intentions mainly aim for porting, testing, and optimizing our approach on a real MIS device. In this context, it should be analyzed which LED channels are indispensable and which can be considered to be redundant in order to provide sufficiently good reconstruction results for achieving significantly enhanced color reproduction properties. So far, the LED selection in the current work was more or less arbitrary with the sole objective of being able to imitate reference illuminant D65 as good as possible. Even though verification is still pending, the blue LED channel in particular is supposed to be superfluous for the intended application of reconstructing the spectral reflectance properties of human tissue and could most likely be omitted in a prospective implementation.

Since the reported results so far were only obtained for *ex vivo* animal tissue samples, a further research question that still remained unanswered but should be addressed in the near future is whether the Wiener filter-based approach implemented for the current work is also suitable to be used for applications involving *in vivo* real human tissue. Due to biological similarity, it is indeed assumed that comparably excellent results could also be obtained in these cases—nevertheless, proof is still necessary and can only be provided after the proposed method has been successfully ported to a feature-complete MIS test device, which, as stated at the beginning of the last paragraph, should therefore be the main focus of further considerations.

## Appendices

## Appendix

As stated in Sec. 2, the Wiener filter matrix $\mathbf{W}$ is intended to minimize the MSE

between the actual $\mathbf{r}$ and the estimated $\widehat{\mathbf{r}}$ reflectance spectrum, where $\u27e8\u2022\u27e9$ denotes the expectation value operator. By applying Eq. (3), this MSE can be rewritten as follows:## Eq. (15)

$$e=\u27e8{\mathbf{r}}^{T}\mathbf{r}-{\mathbf{r}}^{T}\widehat{\mathbf{r}}-{\widehat{\mathbf{r}}}^{T}\mathbf{r}+{\widehat{\mathbf{r}}}^{T}\widehat{\mathbf{r}}\u27e9\phantom{\rule{0ex}{0ex}}=\u27e8{\mathbf{r}}^{T}\mathbf{r}\u27e9-\u27e8{\mathbf{r}}^{T}\widehat{\mathbf{r}}\u27e9-\u27e8{\widehat{\mathbf{r}}}^{T}\mathbf{r}\u27e9+\u27e8{\widehat{\mathbf{r}}}^{T}\widehat{\mathbf{r}}\u27e9\phantom{\rule{0ex}{0ex}}=\u27e8{\mathbf{r}}^{T}\mathbf{r}\u27e9-\u27e8{\mathbf{r}}^{T}\mathbf{Wc}\u27e9-\u27e8{\mathbf{c}}^{T}{\mathbf{W}}^{T}\mathbf{r}\u27e9+\u27e8{\mathbf{c}}^{T}{\mathbf{W}}^{T}\mathbf{Wc}\u27e9,$$Mathematically, the task of finding a minimum of Eq. (15) with respect to $\mathbf{W}$ can be performed by setting its corresponding partial derivative to zero, i.e.,

## Eq. (16)

$$\frac{\partial e}{\partial \mathbf{W}}=-2\u27e8{\mathbf{rc}}^{T}\u27e9+2\mathbf{W}\u27e8{\mathbf{cc}}^{T}\u27e9=0,$$^{60}

## Eq. (18)

$$\frac{\partial {\mathbf{a}}^{T}{\mathbf{X}}^{T}\mathbf{b}}{\partial \mathbf{X}}={\mathbf{ba}}^{T},$$## Eq. (19)

$$\frac{\partial {\mathbf{a}}^{T}{\mathbf{X}}^{T}\mathbf{Xb}}{\partial \mathbf{X}}=\mathbf{X}({\mathbf{ab}}^{T}+{\mathbf{ba}}^{T}).$$Solving Eq. (16) for the Wiener filter matrix gives

## Eq. (20)

$$\mathbf{W}=\u27e8{\mathbf{rc}}^{T}\u27e9{\u27e8{\mathbf{cc}}^{T}\u27e9}^{-1}\phantom{\rule{0ex}{0ex}}=\frac{\u27e8\mathbf{r}({\mathbf{r}}^{T}{\mathbf{\Lambda}}^{T}+{\mathbf{n}}^{T})\u27e9}{\u27e8(\mathrm{\Lambda}\mathbf{r}+\mathbf{n})({\mathbf{r}}^{T}{\mathbf{\Lambda}}^{T}+{\mathbf{n}}^{T})\u27e9}\phantom{\rule{0ex}{0ex}}=\frac{\u27e8{\mathbf{rr}}^{T}\u27e9{\mathbf{\Lambda}}^{T}+\u27e8{\mathbf{rn}}^{T}\u27e9}{\mathbf{\Lambda}\u27e8{\mathbf{rr}}^{T}\u27e9{\mathbf{\Lambda}}^{T}+\u27e8{\mathbf{nn}}^{T}\u27e9+\mathbf{\Lambda}\u27e8{\mathbf{rn}}^{T}\u27e9+\u27e8{\mathbf{nr}}^{T}\u27e9{\mathbf{\Lambda}}^{T}}.$$Under the assumption of dealing with random noise of zero mean and variance $\sigma $, which is independent of the spectral reflectance, the subsequent equations hold for the respective expectation values:

## Eq. (21)

$$\u27e8{\mathbf{rn}}^{T}\u27e9=\u27e8\mathbf{r}\u27e9\underset{\begin{array}{c}=0\end{array}}{\underset{\u23df}{\u27e8{\mathbf{n}}^{T}\u27e9}}=0,$$## Eq. (22)

$$\u27e8{\mathbf{nr}}^{T}\u27e9=\underset{\begin{array}{c}=0\end{array}}{\underset{\u23df}{\u27e8\mathbf{n}\u27e9}}\u27e8{\mathbf{r}}^{T}\u27e9=0.$$Hence, Eq. (20) simplifies to

## Eq. (23)

$$\mathbf{W}={\mathbf{K}}_{\mathbf{rr}}{\mathbf{\Lambda}}^{T}{(\mathbf{\Lambda}{\mathbf{K}}_{\mathbf{rr}}{\mathbf{\Lambda}}^{T}+{\mathbf{K}}_{\mathbf{nn}})}^{-1},$$## Acknowledgments

The authors would like to thank Matthias Szarafanowicz, MSc, from the Technische Universität Darmstadt for encouraging comments and fruitful discussions regarding the writing process of this article. Special thanks goes to the Arnold & Richter Cine Technik GmbH & Co. Betriebs KG for providing the necessary insights into camera technology and video-based MIS techniques. This work was supported by the Project No. 13N13394 (UNILED II) of the Federal Ministry of Education and Research (Germany).

## References

## Biography

**Sebastian Babilon** received his MSc degree in physics from the Technische Universität Darmstadt, Germany (2014). Since then, he has been working at the Laboratory of Lighting Technology in Darmstadt as a research assistant and doctoral candidate. His current work is focused on memory related color perception and the spectral optimization of multichannel LED luminaires for improved observer preference.

**Paul Myland** is a graduate student at the Laboratory of Lighting Technology at the Technische Universität Darmstadt, Germany. He received his BSc degree in electrical engineering from the Technische Universität Darmstadt, Germany, in 2016. In his work, he was concerned with the application of a 3D-LUT approach for digital camera characterization. He was gathering industrial experience as an intern at ARRI Cine Technik in Munich, Germany. He is currently working on methods for a fast and reliable reconstruction of object reflectance spectra.

**Julian Klabes** is a graduate student at the Laboratory of Lighting Technology at the Technische Universität Darmstadt, Germany. He received his BSc degree in electrical engineering from the Technische Universität Darmstadt, Germany, in 2016. In his work, he was concerned with the application of various approaches for the characterization of digital display devices. His current work is focused on algorithms for object recognition based on surface reflectance spectra.

**Joel Simon** is an undergraduate student at the Laboratory of Lighting Technology at the Technische Universität Darmstadt, Germany. As part of a student research project, he developed, built, and characterized the experimental device described in this work, which was used for spectral reflectance measurements and reconstruction.

**Tran Quoc Khanh** is a university professor and head of the Laboratory of Lighting Technology at the Technische Universität Darmstadt, Germany. He graduated in optical technologies and obtained a doctoral degree in lighting engineering from the Technische Universität Ilmenau, Germany. Before being appointed as a professor, he gathered industrial experience as a project manager at Arnold and Richter Cine Technik AG in Munich, Germany. His research interests cover all important aspects of modern lighting technology, including LEDs, colorimetry, mesopic vision, glare, photometry, and color science related topics.