2.1.

Multispectral SWIR Fluorescence Imaging Device for Fluorescence-Guided Surgery

A multispectral SWIR fluorescence imaging system was designed and constructed (Fig. 2). Briefly, tissue is illuminated by a 785-nm fiber-coupled laser (BWF-1-785/55371, B&W Tek) dispersed onto the sample using a ground glass diffuser (DG10-220-MD, Thorlabs, Germany). SWIR fluorescence emission from the sample is collected by a highly sensitive InGaAs camera [$\mathrm{QE}>80\%$ 950 to 1600 nm, NIRvana 640, Teledyne Princeton Instruments Fig. 2(b)] coupled to a SWIR lens ($f=16\mathrm{mm}$, F/1.4, Navitar, Canada). Fluorescence light is sequentially filtered using a six-position filter wheel (LTFW6, Thorlabs, Germany) through six long-pass filters with cut-off wavelengths of 850, 950, 1050, 1150, 1250, and 1350 nm [FELH series, Thorlabs, Germany, Fig. 2(b)]. The system was mounted inside a light-tight enclosure to remove background light. The camera was cooled to $-80°\mathrm{C}$ to reduce thermal noise.

Fig. 2

Multispectral NIR-I/SWIR fluorescence imaging device. (a) Photograph of the custom multispectral NIR-I/SWIR fluorescence imaging system alongside its schematic representation. (b) The spectral characteristics of the device are shown, including the QE of the sensor (purple line, data from Teledyne NIRvana specifications), the emission spectrum of IRDye800CW (adapted from Refs. 10 and 23, black line), and the transmission for the long pass filters (orange lines, data from Thorlabs FELH series specifications).

2.2.

In Vivo Fluorescence Imaging in a Small Animal Tumor Xenograft Model

This study assessed multispectral SWIR FGS in a subcutaneous animal model of neuroblastoma (NB). NB is an aggressive extracranial solid tumor accounting for 8% to 10% of all childhood malignancies and $\sim 15\%$ of all cancer-related deaths in the pediatric population.²⁴ With at least one-third of patients presenting with metastases at diagnosis, NB is one of the most challenging malignancies for pediatric oncologists and surgeons.²⁵ Surgical resection of NB is challenging due to the localization, heterogeneity, and aggressive behavior of the tumor, compounded with the lack of real-time tools able to distinguish malignant tissue from the surrounding healthy tissue. The introduction of FGS in NB would transform surgery by providing an objective, real-time tool to visualize the extent of tumor resection, identify residuals and reliably assess the impact of surgical resection.

The recently developed molecular imaging probe Dinutuximab-IRDye800 was used in this study.²⁶ Dinutuximab-beta (Qarziba), a clinically used monoclonal antibody, is targeted to the disialoganglioside antigen GD2 receptor, a clinically relevant tumor-associated antigen abundantly and ubiquitously expressed on almost all neuroblastic tumors, regardless of tumor stage.^27,28 Dinutuximab-beta was conjugated to IRDye800CW (LI-COR Biosciences), the most used fluorophore conjugated to clinically approved monoclonal antibodies in clinical trials.³ The resulting conjugate will be referred to as “Dinutuximab-IRDye800” throughout the manuscript.

The performance of multispectral NIR-I/SWIR fluorescence imaging was assessed in vivo on a NB subcutaneous mouse xenograft. All experimental animal procedures were approved by the department of biological services and were carried out following local and international regulations. Briefly, human NB cells (LAN-1 cells, $2\times {10}^6$) resuspended in Matrigel ($100\mu \mathrm{l}$, Appleton Woods Ltd, United Kingdom) were injected subcutaneously on the right flank of 6- to 8-week-old athymic nude female mice (CD1-Foxn1nu, Charles River Laboratories). Tumor growth was subsequently measured by calipers. Mice were intravenously injected with $100\mu \mathrm{g}$ (resuspended in $100\mu \mathrm{l}$ of PBS) of Dinutuximab-IRDye800 when the tumor was of an adequate size ($\sim 5\times 6\mathrm{mm}$, time $t=0$). At times $t=24$, 48, 72, and 96 h after injection, one mouse was euthanized, the tumor was exposed, and images were captured using the multispectral NIR-I/SWIR fluorescence imaging device. Two tumor-bearing mice not injected with the dye (negative control) were culled when the tumors reached a humane endpoint, and images were captured using the multispectral NIR-I/SWIR fluorescence imaging device.

2.3.

Spectral Modeling

The published emission spectrum of IRDye800CW is known to be suppressed in the high wavelength region due to the low sensitivity of silicon sensors. The true emission spectrum of IRDye800CW was predicted by reflecting the data book absorption spectrum of IRDye800CW using the Franck–Condon principle. This was manually matched to the SWIR emission spectrum of IRDye800CW measured by Antaris et al.¹⁰ to predict a complete IRDye800CW emission spectrum (Fig. S1 in the Supplementary Material). This spectrum was propagated through the transmission characteristics of the system to predict the measured multispectral image spectrum according to

2.4.

Image Processing

2.5.

Classification of Multispectral Image Cubes to Discriminate Tumor and Non-Tumor Tissue

Classification was performed in MATLAB (2022a, MathWorks). Each pixel in the image cube represents a 6-element spectrum. To visualize the variation between- and within-classes, principal component analysis (PCA) of these spectra was performed. PCA takes an n-dimensional ($n$-variable) dataset and projects it onto n new principal component (PC) axes such that the first axis describes the most variance in the data, and each subsequent axis describes most of the remaining variance. Since most of the variation in the dataset can be visualized using the first few PCs, the remaining PCs can be dropped/ignored, allowing graphical visualization of the dataset in 2D or 3D. Furthermore, dropping the latter PCs removes small variations within the data, potentially reducing noise and improving the performance of classification algorithms.

Pixels were classified using four commonly used spectral classification methods:²⁹ linear discriminant analysis (LDA), $k$-nearest neighbor algorithms (KNN), neural networks (NN), and spectral angle mapping (SAM).

LDA classifies spectra by finding a linear combination of features that maximizes the separation between classes relative to within-class variance in the feature space. KNN algorithms classify spectra by choosing the most frequent classes of KNN data points in the feature space ($k=5$). While LDA assumes linear decision boundaries, the KNN algorithm is non-parametric, so makes no assumptions about the shape of the decision boundaries. LDA also assumes variables are Gaussian distributed. NNs perform classification by passing an input vector, in this case a 9-band spectrum, through a series of artificial neurons, with each neuron outputting some non-linear function of its inputs with some weight that is adjusted during training. The output values of the final layer determine the classification. In contrast to LDA, NN classification does not make assumptions about the distribution of input data nor the shape of decision boundaries. The NN was implemented using a 2-layer feed-forward network, with a sigmoid transfer function in the hidden layer (10 neurons) and a linear transfer function in the output layer, using the MATLAB (MathWorks) “neural network pattern recognition app.”

For LDA, KNN and NN classifiers, the 6-element spectra in the training and test datasets were projected onto PCs determined from the training dataset prior to training/testing the classifier (“PCA-LDA,” “PCA-KNN,” and “PCA-NN,” respectively). The effect of dropping low-variance PCs was also investigated by retaining only the first 4, 5, or 6 PCs for classification. For NN, the 6-element spectra were also tested without projection onto PCs.

SAM calculates the n-dimensional spectral angle (SA) between a target spectrum and a reference spectrum; in this case $n=6$. The reference spectra are the mean spectra per-class within the training dataset; thus 3 spectral angles are calculated for each target spectrum — ${\theta }_{\text{tumor}}$, ${\theta }_{\text{non-tumor-tissue}}$, and ${\theta }_{\text{background}}$. For a simple SAM classification, the minimum of these three angles is taken as the predicted class (SAM minimum angle). Alternatively, the angles may be treated as 3-element feature vectors and classified using LDA or KNN (SA-LDA and SA-KNN, respectively).

In summary, seven classification methods were compared: PCA-LDA, PCA-KNN, SA minimum angle, SAM-LDA, SAM-KNN, NN, and PCA-NN. Classification accuracy was determined using cross-validation, with each of the image cubes being used for training and the remaining three image cubes being used for testing (four permutations). For NN classification, one image cube was used for training, one image cube was used for validation, and two image cubes were used for testing. Presented classification accuracies represent the average over all permutations.

2.6.

Simulating the Effects of Exposure on Classification Accuracy

In the real world, FGS imaging conditions vary considerably [Fig. 1(a)]. Many of these variations, such as changes in working distance, illumination intensity and exposure time, can be summarized as a multiplicative change in the light intensity reaching the detector. In the present study, these changes are collectively referred to as changes of “exposure.” To investigate whether classification approaches are robust to changes in exposure, image cubes were multiplied by an exposure factor, $E$, prior to classification ($E=1$ is equivalent to the un-modified image cubes used to train the classifiers).

2.7.

Comparing Multispectral and Monochromatic Fluorescence Imaging

To compare multispectral fluorescence imaging to standard monochromatic fluorescence imaging (fluorescence imaging using a single emission filter), classification of a single filter image ($640\text{pixels}\times 512\text{pixels}\times 1$ filter) was compared with classification based on an image cube ($640\text{pixels}\times 512\text{pixels}\times 6$ filters).

3.1.

SWIR Fluorescence Imaging Enables High Tumor-to-Background Ratio

To investigate SWIR FGS, a multispectral SWIR fluorescence imaging device was constructed and deployed in a preclinical study. Four mice with subcutaneous NB xenografts were imaged at 24, 48, 72, and 96 h (individuals 1–4, respectively) following the injection of Dinutuximab-IRDye800 and 2 control mice were imaged without Dinutuximanb-IRDye800. The acquired images enabled the construction of band images representing fluorescence collected from 100 nm acceptance bands centered at 900, 1000, 1100, 1200, and 1300 nm (850–950, 950–1050, 1050–1150, 1150–1250, and 1250–1350 nm), allowing the relationship between contrast and wavelength to be investigated (Fig. 3).

Fig. 3

(a) Red-green-blue (RGB) and max-normalized fluorescence band images of tumors in four Dinutuximab-IRDye800 injected mice and one representative tumor-bearing control mouse not injected with the dye. Dashed red line is tumor ROI. Dashed blue line is non-tumor tissue ROI. Dashed white line is the background ROI (shown only for mouse 1 [24h]). (b) Line profiles of fluorescence intensity across the tumor for each wavelength band. The gray region shows the tumor region. (c) Mean spectra from within the ROIs. The shaded areas represent the standard deviation over pixels within each ROI.

The controls show negligible fluorescence intensity (control versus Dinutuximab-IRDye800; $19\pm 5$ versus $960\pm 320$ at $>850\mathrm{nm}$), confirming a lack of autofluorescence in the SWIR region. In the Dinutuximab-IRDye800 injected individuals, signal is observed in the non-tumor tissue region, suggesting off-target binding, and scattering of both the on-target and off-target fluorescence.

3.2.

SWIR Fluorescence Imaging Enables Deep Fluorescence Imaging

After tumor resection, one individual was imaged to assess the background in absence of the tumor [Fig. 4(a)]. Off-target liver fluorescence is clearly visible from beneath the tissue surface. Though this is an undesirable off-target effect, it provided an opportunity to observe the depth imaging capabilities of SWIR fluorescence. The liver was surgically exposed to reveal its true location [Fig. 4(b)], confirming the SWIR images accurately delineated the triangular shape of the organ, even as it was buried beneath muscle tissue.

Fig. 4

SWIR enables fluorescence imaging at depth. (a) RGB color image and $>1150\mathrm{nm}$ fluorescence image of a mouse following tumor resection. Off-target fluorescence of the liver can be seen beneath the surface. (b) Subsequent exposure of the liver confirms the location and shape of the observed sub-surface fluorescence corresponding to liver tissue.

3.3.

Machine-Learning Combined with Multispectral SWIR Fluorescence Imaging Enables Accurate Tumor Classification

We hypothesized that fluorescence emission reaching the detector via scattering in non-tumor tissue would have a different spectrum to fluorescence emission arriving directly from the tumor. To test this, average spectra from within tumor and non-tumor tissue ROIs were plotted [Fig. 3(c)]. Indeed, the spectra are distinct and conserved across individuals (Fig. 5). PCA analysis was performed on spectra from mouse 1 (24 h), then spectra from the remaining mice were projected onto the mouse 1 PCs. The spectra clearly cluster by class, with this clustering conserved across the individuals (Fig. 6).

Fig. 5

Average spectra of tumor tissue and non-tumor tissue show subtle but conserved differences. The mean spectra from within the tumor and non-tumor ROIs were calculated for each individual. The red and blue lines represent the mean of these spectra over the individuals ($n=4$), with the shaded regions representing the standard deviation over individuals. The dotted line represents the predicted fluorescence spectrum of pure IRDye800CW.

Fig. 6

PCA analysis of measured spectra shows distinct conserved clustering for each class. PCA analysis was performed on the spectra from the mouse 1 (24 h). Spectra from the remaining individuals were projected onto the PC axes of mouse 1. Distinct clusters can be observed. The controls do not show any spectra in the region where tumor and non-tumor tissue clusters appear in the fluorescence-injected/non-control individuals.

Classification accuracy was determined using cross-validation, with 1 image cube being used for training and three image cubes being used for testing (four permutations). The ROIs drawn in each image cube contained 954, 886, 984, and 1060 pixel-spectra for individuals 1–4, respectively. Seven classification methods and three normalization approaches were compared (Fig. 7). The best performing method was PCA-KNN with AUC=1 normalization using 4 PCs. Classification was possible with 97.5% accuracy (97.1%, 93.5%, and 99.2% accuracy for tumor, non-tumor tissue and background, respectively). Confusion matrices for this method are shown in Fig. S2 in the Supplementary Material. Classification maps are shown in Fig. 8. Though there is some misclassification around off-target sources of fluorescence (femur and liver), the tumor is well delineated from the surrounding healthy tissue.

Fig. 7

Comparison of seven approaches used for classification of spectra. The methods are described in Sec. 2.4. For each approach, three different normalization methods were used: max normalization, division by the max value; AUC = 1 normalization; and SNV normalization. Classification accuracy was determined using cross-validation. KNN, $k$-nearest neighbor algorithm; LDA, linear discriminant analysis; NN, neural network; PC, principal component; PCA, principal component analysis; and SA, spectral angle.

Fig. 8

Machine-learning enables classification of multispectral SWIR fluorescence images with high accuracy. Classification maps for PCA-KNN with $\mathrm{AUC}=1$ normalization and 4 PCs. Training data is from mouse 1 (24 h). Test data 1–3 are from individuals 2–4 (48, 72, and 96 h).

3.4.

Multispectral SWIR Fluorescence Imaging Enables Exposure-Independent Tumor Delineation

Standard FGS uses a monochromatic fluorescence image captured with a single emission filter. Since any classification based on this image must use an imaging-condition-dependent threshold, the resulting classification is susceptible to errors when imaging conditions change. This can be seen in images captured with different exposures and thresholded to show fluorescence overlays (Fig. S3 in the Supplementary Material). If the exposure time is changed, the apparent size of the tumor changes, resulting in false positives at higher exposures / lower thresholds and false negatives at lower exposures / higher thresholds. Even with machine-learning techniques, classification based on an image acquired with a single emission filter (850-nm LP) is highly susceptible to errors due to changes in the exposure factor [Fig. 9(b)], as these methods ultimately use a threshold to define the class boundaries, albeit one determined statistically based on the training data.

Fig. 9

Multispectral SWIR fluorescence imaging enables exposure-factor-independent tumor delineation. (a) RGB color and fluorescence images of the tumor. (b) Classification with just 1 filter image (monochromatic fluorescence image), changes in exposure factor cause large changes in classification maps. Normalization is not possible with a single wavelength. (c) With a 6-filter image cube, but no normalization, classification is more robust to changes in exposure factor, but still fails to properly demarcate the tumor. (d) With both a 6-filter image cube and AUC = 1 normalization, classification is unaffected by changes in exposure factor (by definition). All classification maps are generated using PCA-KNN ($k=5$ neighbors, $\mathrm{PCs}=1$ in B, PCs = 4 in (c) and (d) trained on mouse 1 (24 h) and tested on mouse 2 (48 h). $E=1$ corresponds to the exposure factor of the training data. Accuracy over all pixels within the three ROIs is shown in the top right corner of each classification map.

Classification based on a multispectral image cube results in some improvement to classification accuracy, but since the major difference between classes remains their absolute intensity (rather than spectral shape), classifiers still rely primarily on intensity information, and consequently, changes in exposure factor continue to cause large errors in classification [Fig. 9(c)]. By normalizing the pixel-spectra ($\mathrm{AUC}=1$), classification is based only on spectral shape (not absolute intensity). Thus, classification accuracy is independent of exposure factor (as the exposure factor is divided out during pixel-spectra normalization), demonstrating the potential of multispectral fluorescence imaging for robust fluorescence delineation in real-world FGS [Fig. 9(d)].

In summary, robust tumor delineation requires multispectral information and appropriate normalization; classification approaches applied to monochromatic images fail, since normalization of these images is not possible (without tricky calibration) [Fig. 9(b)], and classification of multispectral images fails without normalization [Fig. 9(c)].