2.1.1.

Illumination channel

Visible illumination was provided by a white light LED (T7358; Prizmatix), filtered with a short and longpass filter (AT465lp; Chroma and FESH0650; Thorlabs) before being coupled into one arm of a bifurcated fiber (19 Fibre Y bundle, BF19Y2HS02; Thorlabs) using a custom LED-to-SMA coupler (Prizmatix). A laser diode (LP685-SF15 mounted in CLD1011LP; Thorlabs) was coupled into the second arm of the bifurcated fiber via a FC-PC single-mode patch cable (P1-630A-FC-1; Thorlabs) and an adjustable collimator (CFC-5X-B mounted in x-y-z adjustable mount CXYZ1/M; Thorlabs). An objective lens ($40\times /0.65$ Plan Achromat Objective, RMS40X; Olympus) was used to couple light from the output port of the bifurcated fiber into the illumination channel of the PolyScope^® (PD-PS-0095, PolyScope^®; PolyDiagnost). The bifurcated fiber, objective lens, and illumination channel of the PolyScope^® were cage mounted to allow for efficient coupling of light into the illumination channel of the PolyScope^®. A custom 3-D printed mount was made to insert the illumination channel of the PolyScope^® into an adapter and cage mountable plate (AD16F and CXY1; Thorlabs). The illumination power at the distal end of the endoscope was measured with a thermal power meter (A-02-D12-BBF-USB, LaserPoint) and yielded $1.88\pm 0.01\mathrm{mW}$ for the laser line and $0.52\pm 0.01\mathrm{mW}$ for the broadband LED, well below the typically illumination power of a standard endoscope.

2.1.2.

Imaging channel

The image relayed by the PolyScope^® imaging fiber bundle was detected by two SRDAs [Fig. 1(c)]. The visible and NIR SRDAs (VIS: CMV2K-SSM4x4-9.2.10.3, NIR: SSM5x5 5.4.20.8, Imec; Belgium) consist of spectral filters monolithically deposited on a pixel level in a $4\times 4$ and $5\times 5$ grid pattern yielding 16 and 25 spectral bands, respectively, arranged in spectral macropixels [Fig. 1(a)]. Spectral data are thus acquired at the expense of spatial resolution. During the lithography process, spectral filters are deposited directly on the CMOS sensors of compact USB-3 xiQ cameras from Ximea. Each of these cameras weighs less than 30 g, is capable of recording video (90 fps), and performs still image acquisition with integration times ranging from $7.4\mu \mathrm{s}$ to 1 s.^44,45 In this work, integration times ranged between 16 to 980 ms. The SRDAs used here cost $>10000\mathrm{USD}$ each, but have potential to be manufactured for lower cost at high volume since the spectral filters are monolithically integrated on sensor during the lithography process.⁴⁵

An infinity-corrected objective lens (Plan Fluorite $20\times$, UPLFLN20X; Olympus) was used to magnify the image of the fiber bundle, resulting in images of the fiber bundle of $\sim 700$ and 1000 pixels diameter on the visible and NIR SRDA, respectively. Differences in the image size most likely arise from different distances between the objective and tube lens (the infinity space) in the two imaging channels but may also indicate slight misalignment in the optical components. A dichroic beamsplitter (652-nm single-edge dichroic, FF652-Di01; Semrock) was used to direct reflectance and fluorescence light toward a visible and NIR SRDA, respectively. A notch filter (685-nm laser notch filter, ZET685NF; Chroma) was placed in the fluorescence beam path to prevent laser light from saturating the NIR SRDA. Two 100-mm focal length tube lenses (Air-Spaced Achromatic Doublet, AR Coating 350-700, ACA254-100-A, and 650-1050 nm, ACA254-100-B; Thorlabs) were used to image the fiber bundle onto the SRDAs. A corner-mounted broadband mirror (21015; Chroma) was included in the reflectance channel for ease of alignment and to increase system compactness. The fluorescence and reflectance channels of the endoscope were separately aligned using an external light source (OSL2 with OSL2BIR bulb; Thorlabs) and the dichroic beamsplitter removed during alignment of the fluorescence imaging channel. The entire system was mounted in a black box (TB5 Black Posterboard, XE25L225/M, XE25L375/M Construction Rails and XE25W3 Quick Corner Cube; Thorlabs) on an optical breadboard (MB4545/M; Thorlabs) for ease of transport, yielding a system of $45\times 60\times 25\mathrm{cm}$ dimension. Each multispectral detector, typically the bulkiest element, measures only $26.4\times 26.4\times 24.6\mathrm{mm}$ and weighs less than 30 g.

2.2.1.

Analysis of fluorescence multiplexing capability

To predict the relative fluorescence signal strength produced from a fluorescent dye placed in the sample plane, the quantum efficiency (${\mathrm{QE}}_{\text{dye}}$), the extinction coefficient (${ϵ}^{+}$), and the spectral overlap between the endoscopic illumination and the excitation spectra of several dyes were considered. The predicted fluorescence signal from six fluorescent dyes (AF647, AF660, AF680, AF700, AF750, and AF780; Invitrogen) was assessed using both published reference spectra (Invitrogen) and measured reference spectra of the dyes dissolved in phosphate-buffered saline (PBS, 10010015; Thermo Fisher), recorded on a plate reader (CLARIOstar; BMG LABTECH).

Relative fluorescence emissions were predicted for each of the dyes [$I_{\mathrm{em}}(\lambda )$] using

2.2.2.

Modeling endmembers for spectral unmixing

To derive useful information from the spectral data cube, it is necessary to relate the spectral data to the chemical composition of the sample. Such relationships can be established via spectral unmixing. When performing MSI, quantification is often achieved using linear spectral unmixing methods⁴⁰ to resolve the relative contribution of different spectral components to the measured signal. Standard linear spectral unmixing methods require that spectra of components contributing to the signal be known a priori and input into the spectral unmixing algorithm. Independently measured reference spectra are often used for this purpose, however, this needs to be adjusted to account for the spectral transmission characteristics of a given system. Measured reference spectra for all chemical components under investigation in our experimental studies were propagated through the model to generate these adjusted endmembers. The calculation was repeated for illumination of a uniformly reflecting target to predict the spectrum of pure reflectance light that would be sampled by the visible and NIR SRDAs.

2.2.3.

Experimental evaluation of model accuracy

The transmission spectrum of the complete endoscope illumination channel was measured by coupling a stabilized broadband light source (SLS201; Thorlabs) to the input port of the bifurcated fiber in place of the white light LED. The broadband light transmitted to the distal end of the PolyScope^® was measured with a spectrometer (AvaSpec-ULS2048-USB2-FCDC; Avantes). The spectral transmission characteristics of the illumination channel of the endoscope were extracted by dividing the measured spectrum by the known spectrum of the stabilized broadband light source and the root-mean-square-error (RMSE) between the normalized modeled and experimentally measured transmission spectra was calculated.

The transmission spectrum of the complete endoscope imaging channel, including SRDAs, was measured by illuminating the distal tip of the endoscope via an optical fiber (M71L01; Thorlabs). Broadband light (OSL2 with the OLSB2 bulb for the reflectance channel and the OSL2BIR bulb for the fluorescence channel; Thorlabs) was passed through a monochromator (CM110 1/8m; Spectral Products) and wavelength scans between 400 and 900 nm in 3-nm increments were performed using LabVIEW^® to control the monochromator and cameras. The integration times of the visible and NIR SRDAs were set to 16 and 60 ms, respectively. The gain level of both cameras was set to 0.0 and 10 image frames were acquired at each wavelength increment. Before and after each scan, 20 dark frames with matched integration times and gain were captured to allow for dark subtraction of the acquired data.

The acquired data were averaged, dark subtracted, and converted to spectral response curves, as described previously.⁴⁰ The response of the SRDA, as measured in digital numbers [DN($\lambda$)], was taken as the average DN of a region of interest (RoI) drawn over the endoscope field-of-view (FoV). These experimentally measured spectral response curves for each SRDA arm were normalized to the highest peak intensity recorded in the reflectance and fluorescence imaging channel, respectively, and compared to the spectral response predicted by the model. The RMSE is reported as the $\text{mean}\pm \text{standard}$ deviation across all spectral bands in each imaging arm to evaluate the accuracy of the model.

To evaluate the accuracy of the modeled endmembers, $40\mu \mathrm{M}$ solutions of the three fluorescent dyes (AF647, AF660, and AF700) in PBS were endoscopically imaged. Thirty microliters of each dye solution was placed in separate wells of a microwell plate (18 well, microslide, 81826; ibidi). The well plate was placed 10 mm under the distal tip of the endoscope. Two replicates of each dye solution were imaged along with two control wells containing $30\mu \mathrm{L}$ of PBS alone. The endoscopically measured reference spectra were extracted from 60 pixel radii circular RoIs placed over the wells of the well plate located using data acquired in the reflectance channel of the endoscope. Reference emission spectra of each of the three dyes, prepared at $60\mu \mathrm{M}$ concentration, were acquired with a plate reader (CLARIOstar; BMG LABTECH) and used to model endmember spectra. The RMSE of the modeled fluorescence spectra was subsequently evaluated by comparison with the experimentally acquired RoI spectra.

2.4.1.

Spatial resolution

The endoscope spatial resolution (limited by the imaging fiber bundle) was determined at two WDs by imaging a 1951 United States Air Force (USAF) test chart target (#53-714; Edmund Optics) with the visible SRDA only. The spatial resolution was determined by the Michelson contrast from individual spectral band images from the visible MSI data cube according to

This was obtained by fitting a sinusoidal curve over the line pattern of a resolution element of the 1951 USAF target and extracting the maximum ($Y_{\max }$) and minimum ($Y_{\min }$) intensity from the fitted sine curve. Endoscopic illumination nonuniformities, common to endoscopic imaging⁶ arising from effects, such as vignetting, required that a polynomial baseline correction of the cross-sectional intensity profile was applied prior to extracting the Michelson contrast. Images were acquired using the internal illumination of the endoscope at WDs of 5 mm ($\text{camera integration time}=480\mathrm{ms}$ and $\text{gain}=0$) and 10 mm ($\text{camera integration time}=900\mathrm{ms}$ and $\text{gain}=0$).

2.4.2.

Barrel distortion and field-of-view

To characterize barrel distortion in the acquired images, a checkerboard of 1 mm square pattern was imaged at WDs between 2.5 and 15 mm in 2.5-mm increments. The checkerboard pattern was illuminated with an external light source (OSL2 with OSL2 bulb; Thorlabs) and three images from the MSI data cube of the visible SRDA ($\text{peak}\text{wavelength}=462$, 566, and 638 nm) were used for the evaluation using previously published methods.⁴² We identified the position of the center of the fiber bundle and plotted the radial distance to each vertex in the checkerboard pattern $r_d$ (in pixels) against the true distance $r_u$ (in mm) known from the dimensions of the checkerboard pattern.

When imaging a flat target in air, the effective FoV of the PolyScope^® is limited by the extent of the illumination rather than the optics of the imaging channel. The angular extent of the illumination cone was defined as the area within which the diffuse reflectance in all spectral bands in the visible SRDA exceeded 10% of the maximum reflected intensity. The diffuse reflectance was determined by acquiring images of a laminated white paper at WDs incremented from 2.5 to 15 mm in 2.5-mm increments.

To determine the angular FoV of the imaging channel, the protocol was repeated but with external illumination (OSL2 broadband light source with OSL2B bulb; Thorlabs). The imaging FoV of the endoscope was defined as the area within which the pixel values were larger than zero across all spectral bands in the visible SRDA. To more closely mimic endoscopic operation in the GI tract, barrel distortion and FoV were also measured under the following conditions: with water droplet added to the distal tip and with both the distal tip and imaging target fully submerged in water. The pixel radii of the circular FoVs were converted to mm, accounting for the barrel distortion at each WD. The angular FoV of the endoscope was extracted from the tangent of the gradient of the linear fit of $r_{\mathrm{FoV}}$ plotted against WD. Errors were calculated by addition of the fit standard error (one-sigma confidence interval) in quadrature. The camera integration times varied between 30 ms and 1 s according to the WDs and scenes imaged.

After these characterization steps, all subsequent imaging was performed at a 10 mm WD, as this gave a sufficient FoV and spatial resolution for our initial imaging performance evaluation of the spectral endoscope.

2.4.3.

Spectral fidelity

To characterize the uniformity of the spectral response over the endoscopic FoV, we defined a “spectral fidelity metric,” which expresses the MSE of a normalized pixel spectrum compared to that of the average spectral response across the FoV when imaging a uniformly reflective target. The spectral fidelity in the visible SRDA was determined by imaging ($\text{camera integration time}=995\mathrm{ms}$ and $\text{gain}=0$) a uniformly reflective target (Lambertian White Screen, SG3151-0; Sphere Optics).

2.5.1.

Phantom preparation and imaging

To evaluate the spectral imaging performance of the endoscope, we developed an agarose phantom containing two capillaries of chemically oxygenated and deoxygenated mouse blood and fluorescent dye inclusions of AF647, AF660, and AF700 dissolved in PBS. The phantom was designed such that the blood capillaries and the fluorescent dyes could be visualized within the same endoscopic FoV at a 10-mm endoscopic WD. The visible camera integration time was set to 980 ms and its gain to 0, while the NIR camera integration time was 980 ms with gain 5.

The phantom base material was prepared according to a previously published recipe⁴⁰ and poured into a 120-mm diameter glass petri dish to form a 5-mm thick agarose layer. Four holes were made in the agarose slab using a transparent straw with 3-mm internal diameter (391SIPCL; Plastico). Four 5-mm long pieces of transparent straw, of which one side was sealed with a glue gun (PA6-GF30; Type PX 06; Henkel Pattex Supermatic), were placed in the holes with the glue-sealed end facing down. Each straw was filled with a $80\text{-}\mu \mathrm{M}$ pure fluorescent dye solution of AF647, AF660, or AF700 dissolved in PBS and one hole filled with a PBS control.

Mouse blood was obtained from seven female C3H/HeOuJ mice (6 to 10 weeks old) after euthanasia by exposure to raising ${\mathrm{CO}}_2$ concentration. One minute after the complete cessation of respiratory movements, about 0.5 mL of blood was collected by cardiac puncture from each mouse and placed in a microtube containing $10\mu \mathrm{L}$ of heparin sodium salt. Animal handling and euthanasia complied with the British schedule 1 of the Animals (Scientific Procedures) Act 1986. The mouse blood was chemically deoxygenated following a protocol developed by Briley-Sæbø and Bjørnerud.⁴⁷ Deoxygenation was achieved by adding, and thoroughly mixing, 1.5 mg of sodium dithionite (157953-5G-D; Sigma Aldrich) with 0.5 mL of heparinized blood. Successful deoxygenation of the blood was verified with a dissolved oxygen monitor (OxyLite with NX/BF/OT/E probe; Oxford Optronix) on which the deoxygenated blood had a partial oxygen pressure below the detection range of the sensor. Blood was chemically oxygenated by adding $1\mu \mathrm{L}$ of 30% hydrogen peroxide (216763-100ML; Sigma-Aldrich) to 0.5 mL of blood. The partial oxygen pressure in the oxygenated blood saturated the dissolved oxygen monitor, indicating a partial pressure higher than 200 mmHg. Deoxygenated and oxygenated blood were subsequently pulled into separate 100-mm glass capillaries (CV1518; CM Scientific) via capillary action. The capillaries were sealed using capillary tube sealant (02678 Fisherbrand Hemato-Seal Capillary Tube Sealant; Fisher Scientific). Reference spectra of the blood capillaries and the phantom base material were acquired with a calibrated bifurcated reflection probe (RP29, Reflection Probe with Linear Leg; Thorlabs) and spectrometer (AvaSpec-ULS2048-USB2-FCDC; Avantes). Successful oxygenation and deoxygenation was further verified by qualitative comparison of these spectra with tabulated literature data.⁴⁸ A reference spectrum of the endoscope white light LED source was obtained by placing the reflection probe flush against the reflectance target. The phantom was imaged three times using the multispectral endoscope, repositioning the phantom between each repeat.

2.5.2.

Phantom data analysis

Endmember spectra for the fluorescent dyes were obtained by propagating the reference fluorescence spectra, previously acquired on the plate-reader, through the spectral transmission model. Endmember spectra for spectral unmixing of the oxygenation status of the blood were obtained by propagating the acquired reference reflectance spectra [Fig. 4(a)] through the spectral transmission model. The propagated reflectance spectra of the blood capillaries ($I_{\text{blood}}$) and Lambertian white screen ($I_{\text{white}}$) were used to calculate optical density (OD) spectra:

Fig. 4

Calculation of endmembers for spectral unmixing of reflectance and fluorescence imaging data using the spectral transmission model. (a) Reference reflectance spectra measured from glass capillaries containing oxygenated and deoxygenated blood, and a “blank capillary” containing only PBS. (b) The corresponding optical density (OD) spectra calculated from the experimentally measured reference spectra shown in panel (a) compared to literature values (dashed lines).⁴⁸ (c) Comparison of predicted endmembers (solid line) and the endoscopically measured spectra of the oxygenated and deoxygenated blood (points). (d) Comparison of predicted endmembers (solid line) and the endoscopically measured spectra of the fluorescent dyes (points). Max–min normalization was performed on the predicted endmembers and the endoscopically measured spectra.

The log of the spectral band images acquired in the reflectance channel of the endoscope, divided by the spectral band images of the reflectance target, was calculated to obtain the OD of the MSI reflectance data cube prior to spectral unmixing.

3.2.1.

Angular field-of-view

The FoVs defined by the extent of the light cone emerging from the illumination channel of the endoscope were measured to be $58\pm 3\mathrm{deg}$, $65\pm 4\mathrm{deg}$, and $52\pm 1\mathrm{deg}$ when operating: in air [Fig. 5(a)], with a droplet on tip, and fully submerged in water, respectively. The respective FoVs defined by the imaging channel (measured using external illumination) were $102\pm 8\mathrm{deg}$, $96\pm 5\mathrm{deg}$, and $84\pm 5\mathrm{deg}$. As expected, in both cases, the FoV is reduced when the endoscope operates immersed in water, due to the change in refractive index. Within experimental uncertainties, the FoV of the imaging channel when operating in air agrees with the supplier specified FoV of $120\pm 10\mathrm{deg}$.⁴⁹ Differences between imaging of a flat target and within a tubular structure (according to the intended use) may explain the discrepancies between the FoV of the illumination and imaging channel,⁴⁹ which were here measured using a flat target.

Fig. 5

Technical characterization of the endoscope performance. (a) The FoV of the endoscope when operating in air (as defined as the angular extent of its illumination cone) was measured at a range of WDs, from which an angular FoV of $58\pm 3\mathrm{deg}$ was determined. (b) To extract the radius of the FoV at a specific WD, a checkerboard pattern was first imaged to characterize the barrel distortion. The barrel distortion was extracted by fitting the true radial distance from the centre of the FoV to each vertex in the checkerboard pattern $r_d$ (in pixels) against the true distance $r_u$ (in mm). (c) The spatial resolution of the endoscope at WDs of 5 and 10 mm was determined by imaging a USAF test chart target and plotting the Michelson contrast against the number of line pairs/mm. (d) “Spectral fidelity” was calculated spatially to assess the uniformity of the spectral response across the endoscopic FoV. The spectral fidelity map shows the MSE of the normalized spectrum in each pixel compared to the FoV average.

3.2.2.

Barrel distortion

For each WD, the barrel distortion of the endoscope was characterized by imaging a checkerboard pattern and plotting the radial distance to each vertex in pixels ($r_d$) against the true distance ($r_u$) in mm, for three spectral bands in the visible SRDA [Fig. 5(b)]. In line with expectation, no distinctive wavelength dependence could be observed and data from the three spectral bands were therefore grouped in the analysis. The radii of the FoVs of view measured in pixels were converted into mm, taking the effect of the Barrel distortion into account. Due to the limited Barrel distortion observed, no barrel distortion correction was performed during image processing.

3.2.3.

Spatial resolution

The Michelson contrast for a set of resolution elements of 1951 USAF test chart target was evaluated at endoscopic WD of 5 and 10 mm to determine the spatial resolution of the endoscope [Fig. 5(c)]. A linear fit was applied to the data and the spatial resolution was extracted based on a Michelson cut-off contrast of 10%. The smallest resolvable line pairs/mm were $1.7\pm 0.6$ and $1.3\pm 0.2$ at 5 and 10 mm WD, respectively; this corresponds to spatial resolutions of $300\pm 100\mu \mathrm{m}$ and $400\pm 100\mu \mathrm{m}$. The uncertainty was calculated by adding the standard errors of the fit parameters (one sigma confidence interval) in quadrature.

3.2.4.

Spectral fidelity

The uniformity of the spectral response across the endoscopic FoV was characterized by imaging a uniformly reflecting target. “Spectral fidelity,” evaluated spatially, is calculated as the mean-squared-error (MSE) of a given pixel spectrum compared to the average spectral response across the FoV. A high “spectral fidelity” (i.e., low average MSE of $0.0064\pm 0.0070$) was observed [Fig. 5(d)]. Some degradation in the spectral fidelity could be observed toward the edges of the endoscopic FoV, which may arise from slight chromatic aberrations or a decrease in the performance of the honeycomb correction toward the edges of the FoV. The nonsymmetric nature of the degradation further suggests a slight tilt in one of the optical components in the imaging path. Nonetheless, based on the small overall difference in the spatial MSE, we can assume that such effects are small and would not be expected to impact spectral unmixing.