Tissue interrogation by means of optical fluorescence imaging is one of the most widespread methods in medicine and biology.¹ Recently, molecular fluorescent agents have emerged as a means to enhance the human vision by targeting disease biomarkers.² Fluorescence imaging with molecular specificity has been successfully translated into clinical applications to enable the accurate cancer identification and demarcation^3,4 as well as sentinel lymph node localization.^5,6

Intraoperative imaging systems such as the ones presented in Refs. 7 or 8 typically capture the fluorescence emission waveband as well as an additional color image. In order to provide the surgeon with easy and intuitive orientation that matches his field of view, the detected fluorescence signal is superimposed on the color image.

Typically, a pseudocolor image is generated by superimposing the fluorescence image over the color image in lime green using alpha blending.⁹ A transparent overlay can be used to maintain the structural information present in the color image while adding the molecular information of the fluorescence distribution.

Biological data visualization requires standardization and robustness to be intuitively interpretable and easily transferable. However, a transparent green overlay is not robust, since it is based on the presence of a predominantly redish tissue background. Thus, in the cases where this assumption is violated, it will produce low contrast and an inappropriate visualization. In this work, we propose alternative schemes to select the superimposed color in an adaptive way that does not make a priori assumptions on the sample’s color appearance. Furthermore, we showcase a novel visualization method for video-rate surgical fluorescence imaging that modulates transparency and color over the time.

Before the two acquisition channels can be merged into a composite image, a number of preprocessing steps are required. Depending on the system setup, resizing of the fluorescence channel and coregistration with the color image, e.g., by an affine transformation, might be necessary. In order to restrict the superimposed fluorescence signal to physiologically relevant information, denoising and thresholding can be applied to remove autofluorescence, filter cross-talk and other parasitic signals.¹⁰ An appropriate threshold can be determined either manually, or by an automatic thresholding or segmentation algorithm.¹¹ Herein, we used Otsu’s method¹² to determine the threshold $T$, for which the preprocessed fluorescence image ${\mathbf{F}}^{\prime }$ is then calculated from the measured intensity image $\mathbf{F}$ as

The composite image was evaluated by computing the color contrast $\mathrm{\Delta }{E}_{00}$ between the superimposed fluorescence region and the surrounding area in the overlay using the CIEDE2000 formula.¹³ Thereby, a high value indicates that the fluorescent region is colored in a way that can easily be distinguished from the nearby regions.

Alpha blending combines the two images by weighting individual pixels with the transparency value $\alpha$.⁹ Blending the fluorescence image ${\mathbf{F}}^{\prime }$ and the color image $\underline{\mathit{C}}$ with RGB channels ${[\begin{array}{ccc}{\mathit{C}}_{\mathrm{R}}& {\mathit{C}}_{\mathrm{G}}& {\mathit{C}}_{\mathrm{B}}\end{array}]}^{\mathrm{T}}$ is given as

Thereby, $\mathit{P}$ is the composite image, $\alpha \in [0;1]$ is the transparency, and $\underline{O}$ is the pseudocolor of the overlay. As the default color, we selected the lime green, i.e., $\underline{O}={[\begin{array}{ccc}0& 1& 0\end{array}]}^{\mathrm{T}}$, for the superimposition on biological tissue. Additionally, we applied a pixel-wise variable map for the transparency to obtain smoother transitions between areas above and below the fluorescence threshold. For a maximum transparency ${\alpha }_m$, this can be written as

An adaptive approach to select a pseudocolor $\underline{O}$ that yields a strong contrast is based on picking a color that is underrepresented in the region of interest (ROI). The color image was transformed into the cylindical-coordinate HSV represenation (Hue, Saturation, Value),¹⁴ and the average hue $H_{\mathrm{avg}}$ was calculated in the ROI where the fluorescence lies above the threshold. The hue of the overlay color was chosen to lie on the opposite side of the HSV cone, while both the saturation and value were set to 1

The obtained color was transformed back into the RGB space and utilized for alpha blending as described above.

Alternatively, robust overlay was done by principal component analysis (PCA)¹⁵ of the color image, where the data are projected onto a set of linearly uncorrelated base vectors. Thereby, the ROI was vectorized and PCA was performed to yield the principal component loadings $\mathit{U}$. The transformation of the color image into the coordinate system spanned by the principal component eigenvectors is given by ${\underline{\mathit{C}}}_{\mathrm{PCA}}={\underline{\mathit{U}}}^{\mathrm{T}}\underline{\mathit{C}}.$

As the principal components are ordered by their statistical contribution to the original image, we superimposed the preprocessed fluorescence image on the third, least significant component in order to achieve the highest contrast to the main image features

The composite image was then transformed back into the original RGB color space by multiplying with the orthonormal matrix $\underline{\mathit{U}}$

Thus, blending the fluorescence over the least significant principal component is equivalent to selecting the eigenvector with the smallest eigenvalue as the overlay “color” for alpha blending in RGB space.

The methods discussed so far aim to integrate the fluorescence information into the spectral components of the color image, generating the contrast by selecting a color that is underrepresented in the image. Alternatively, especially in the case of continuously acquired clinical images, the signal can also be visualized along the temporal dimension. It was assumed that the image changes slowly compared to the acquisition frame rate, hence changes occurring with a higher temporal frequency are easily identifiable in the overlay. The temporal alpha blending can then be written as

A sinusoidal curve was used to vary transparency with a frequency of 2 Hz and the overlay color hue with 1 Hz.

Color and epi-fluorescence images from a variety of biological studies as shown in Fig. 1 were chosen to test and validate the proposed algorithms. Image set A stems from a study of leaves transpiration and vasculature network, where the leaf absorbed Alexa Fluor 750 through its petiolule. The images of sets B and C were acquired using a prototype cryoslicer-based imaging system.¹⁶ Set B shows an axial section of a mouse bearing a tumor targeted by a virus that expressed green fluorescent protein (GFP) and melanin.¹⁷ In image set C, we depict two zebrafish with pan-neuronal expression of GFP embedded in the same block. In set D, we imaged a near-infrared probe (cetuximab-CW800) targeting intraperitoneal tumors (HT29luc2) through the intact skin of a nude mouse.¹⁸ In dataset E, we simulated an intraoperative scenario using a subcutaneous 4T1 tumor targeted with bevacizumab-CW800.

Color images acquired from the five biological samples are displayed in Figs. 1(a)–1(e), while the corresponding fluorescence channel is shown in Figs. 1(f)–1(j). The mouse images contain parasitic signals from intestinal autofluorescence [Fig. 1(g)], skin autofluorescence [Figs. 1(i) and 1(j)], and contrast agent accumulation in the liver [Fig. 1(i)]. The images were coregistered and automatic segmentation of the fluorescence signal was performed using Otsu’s method. The ROI was defined for each image as the area containing fluorescence signal above the threshold.

Overlay images were calculated for all the datasets using the pseudogreen alpha blending, color picking in HSV space, and the PCA-based alpha blending. Figures 2(a)–2(m) show the overlay in lime green, which perform best in the case of a predominantly red background [Fig. 2(m)]. The color selection in HSV space is plotted in Figs. 2(f)–2(j). The algorithm picks various shades of cyan for the animal tissue and an orange color for overlay on the leaf. Color selection using PCA is shown in Figs. 2(k)–2(o) and yields a wider range of colors for the different datasets. The time-modulated overlay is shown in Video 1.

The average value of $\mathrm{\Delta }{E}_{00}$ over the five datasets was computed to be 19.7 for the lime green, 20.5 for the HSV method, and 23.0 for the PCA method, respectively. The two adaptive methods provide a good average color contrast, yet it is notable that the HSV method yielded an unsatisfactory result for dataset C, while the PCA selection seems more consistent in selecting an appropriate overlay color.

The ongoing translation of fluorescence molecular imaging into clinical and surgical practice calls for standardized and robust ways to represent multispectral data. Fluorescence emission wavebands contain molecular information on tissue that is invisible to the human eye, while additional color images provide anatomical information and easy orientation. In this work, we investigated the methods for the fusion of multispectral fluorescence data into a composite RGB image.

Schemes for the generation of a pseudocolor fluorescence overlay were discussed. Among those, the overlay in lime green is widely used for the biological tissue, but it is based on a priori assumptions about the spectral characteristics of the imaged sample. Therefore, we proposed more robust alternatives that utilize hue or PCA to adaptively select a color for the overlay, based on the tissue characteristics. Additionally, visualization along the temporal dimension of color and fluorescence data acquired in real time was proposed and demonstrated.

The problem can be generalized to the representation of an $n$-dimensional dataset in the RGB color space. Herein, we assumed a single fluorescence measurement that is superimposed on a corresponding color image. Such a projection onto a lower dimensional space is inherently lossy; however, the results indicate that the adaptive methods can yield an improved average color contrast. Given the wide range of biological images considered, future work should study the performance of the presented methods on a bigger representative dataset or for individual fields of application. Furthermore, the performance evaluation could be conducted using questionnaires to physicians instead of the contrast formula employed herein.

As an extension to the presented cases, measurements might be taken with multiple fluorochromes or with just a single visible wavelength for orientation. Modified version of the current algorithms for novel schemes need to be developed to provide even more flexibility under such a wider range of conditions. One possible approach could be the utilization of an overlay color that varies pixel-wise or within the image region to account for images with a greater color variation.