1.

Introduction

Diffuse optical imaging using either temporal or spatial modulation of light has shown strong promise for quantitatively assessing functional parameters in living tissue, such as content in oxyhemoglobin, deoxyhemoglobin, water, and lipids. Such parameters can be used to determine the physiological condition of the tissue and provide critical information to healthcare professionals for decision making. For instance, temporal frequency domain monitoring of breast tissue during neoadjuvant chemotherapy treatment can help differentiating responding versus nonresponding patients within 10 days of treatment.^1,2 In other studies, spatial frequency domain imaging (SFDI) has demonstrated the potential to diagnose patients early with diabetic foot ulcers, providing means to adapt treatment toward limiting morbidity and decreasing healthcare burden.^3,4

While these methods have demonstrated a strong potential to aid in decision making and solve concrete clinical problems, their adaptation to challenging clinical environments has been very limited. Surgery is one of these environments where information must be provided quantitatively (i.e., repeatable and interpretable), in real time (i.e., $>25$ images per seconds), and over a large field-of-view (i.e., $>10\times 10\mathrm{c}{\mathrm{m}}^2$).⁵ To attain this challenging objective, our group has been developing solutions based on SFDI. Because SFDI works in the frequency domain, it allows the capture of information for all the pixels of an image at once, providing a significant advantage over raster-scanning methods. The original developments of SFDI relied on the acquisition of several images at single wavelengths, providing quantitative multipixel optical properties measurements.^6–8 More recent developments have introduced near-real-time [e.g., 10 frames per second (fps)] and real-time (i.e., $>25\mathrm{fps}$) imaging methods such as single snapshot of optical properties (SSOP).^9–11 These methods process the acquired information directly in the frequency domain offering the opportunity to drastically reduce the acquisition time down to a single acquired image for obtaining optical properties maps of living tissues. However, these methods still acquire only one wavelength at a time, making them suboptimal for imaging living tissue parameters (e.g., chromophores concentration) in real time.

In order to deploy such technology in the operating room, methods capable of performing multi- or hyperspectral imaging quantitatively, over a large field-of-view and in real time, must be developed. Recent work has proposed dual-wavelengths imaging for extracting blood parameters such as the concentration of oxyhemoglobin, deoxyhemoglobin, or oxygenation. For instance, SSOP was used in conjunction with speckle imaging using a scientific CMOS camera by sequentially acquiring wavelengths, leading to 16 fps imaging.¹² However, this work uses a mechanical optical chopper and low-power (150 mW) laser diodes, making it difficult to achieve higher frame rates and limiting its scalability to a higher number of wavelengths. Another, more recent work used square patterns to improve switching time using a standard digital micromirror device (DMD) to project three wavelengths sequentially at three phases (standard SFDI acquisition), and compressed single pixel imaging leading to 12.5 fps imaging.¹³ However, relying on sequential frames to acquire optical properties maps naturally restricts the possibility to achieve higher frame rates. As of today, because of these instrumental limitations, no method has been developed for real-time (i.e., $>25$ images per seconds), wide-field (i.e., $>10\times 10\mathrm{c}{\mathrm{m}}^2$), and quantitative imaging for clinical applications.

In this work, we present a method using both spatial and temporal modulation of light to provide images interpretable by clinicians in time-constrained environments such as surgery. We propose here a method where sources are modulated sinusoidally in time using signal generators (no mechanical chopper) and projected in the form of single-frame patterns for processing using the SSOP method (no need for sequential frames). Sources are modulated in time to encode wavelengths and combined in a multifiber bundle connected to a spatial-light modulator. Light is then projected with specific patterns onto the field-of-view and images captured using a fast camera. The obtained three-dimensional (3-D) cube of data (two-dimensions + time) is then processed in the temporal frequency domain to separate wavelengths, and the resulting images are processed in the spatial frequency domain using the SSOP methodology to extract optical properties for each wavelength and then calculate blood parameters (oxyhemoglobin, deoxyhemoglobin, and oxygenation). In the first part of this paper, we present the imaging method, processing, and hardware. Then, we validate the method in comparison with standard, non-real-time SFDI onto tissue-mimicking phantoms. Finally, we perform a validation on a living hand during an arm-cuff occlusion. Altogether, this study contributes to the translation of endogenous diffuse optical imaging for surgical guidance.

2.

Material and Methods

2.1.

Spatiotemporal Modulation of Light

The concept of multispectral imaging using temporal modulation of light sources for wavelength encoding is not new, nor are multispectral spatial frequency domain methods.^13–18 However, until now, imaging living tissue parameters over large fields-of-view at high resolution, quantitatively, and in real time remained a significant challenge. The principle underlying the spatiotemporal modulation of light method is shown in Fig. 1. Discrete sources are modulated in time, each wavelength being at a different specifically chosen temporal frequency. The light outputs from all sources are combined using a multifiber bundle connected to a spatial light modulator. Patterns of light combining all temporally modulated wavelengths are projected onto the field-of-view and a fast camera is used to acquire the scene. The acquired images are first processed using a temporal discrete Fourier transform (DFT) to accurately separate wavelengths contributions. Finally, the resulting images are processed in the spatial frequency domain using SSOP methodology to obtain the optical properties at the chosen wavelengths.

Fig. 1

Spatiotemporal modulation of light: acquisition and processing. Sources at different wavelengths are modulated in time at specifically chosen frequency. Light from these sources is combined and modulated in space using a spatial light modulator. The scene is acquired using a fast camera. The acquired images are then processed in the temporal frequency domain to separate the contribution from the different light sources, and processed in the spatial frequency domain to extract the optical properties at all wavelengths.

One of the significant hurdles in implementing such method is in ensuring that wavelengths are precisely and correctly modulated, sampled, and processed to unambiguously extract the information content contained in each temporally modulated source. Not performing these operations correctly would result in inaccuracies in the extracted optical properties and therefore in the assessment of tissue parameters. Two strategies can be used to perform these measurements: either a fine temporal sampling of the scene allowing to use a generic fast Fourier transform (FFT) function to separate any temporal frequency contribution, or a custom DFT function allowing to sample only a chosen number of frequencies. While a generic FFT may be more suited to analyze a large amount of contributions, it can be more costly in terms of acquisition and processing time compared with a custom DFT. In this work, because we focus on acquiring real-time images, we chose to implement a custom DFT to allow ultrafast computation of the chosen sources’ contributions. The DFT of one pixel in time is given as

Eq. (1)

$$X(k)=\sum _{n=1}^{N}x(n).e^{-\frac{j2\pi k(n-1)}{N}},$$

and the proper frequencies that should be used to modulate are exact frequencies in the discrete Fourier domain, for instance, in the case of four sources:

Eq. (2)

$$f_1=\frac{k_1.F_e}{N};f_2=\frac{k_2.F_e}{N};f_3=\frac{k_3.F_e}{N};f_4=\frac{k_4.F_e}{N},$$

with $F_e$ being the sampling frequency, $N$ is the number of acquired images, and $k_i$ determined such that the captured signal spans an integer number of periods.^19,20

If these conditions are gathered, on top of obvious considerations such as avoiding harmonics or standard electrical current frequencies, and along with a precise synchronization of the camera and signal generators, the contribution of each source can be precisely and unambiguously determined.

3.

Results

3.1.

Phantom Experiments

Results from the phantom experiments are summarized in Fig. 2. On average, optical properties obtained using spatiotemporal modulation of light show good agreement with the standard SFDI method. Mean errors were calculated for all the pixels in a $30\times 30\mathrm{m}{\mathrm{m}}^2$ region taken in the middle of each phantom. For 665 nm, the mean error is $1.8\%\pm 1.2\%$ in absorption and $3.3\%\pm 1\%$ in reduced scattering. For 860 nm, the mean error is $4.2\%\pm 2.1\%$ in absorption and $2.5\%\pm 1.1\%$ in reduced scattering.

Fig. 2

Results from phantoms experiments. Absorption and reduced scattering are shown (mean and standard deviation) over a wide range of optical properties. Results are shown for both wavelengths using standard SFDI as a reference and spatiotemporal modulation of light.

Actual images of four phantoms are shown in Fig. 3 for both wavelengths, in absorption and reduced scattering. Results from standard SFDI are shown, as well as spatiotemporal modulation of light, a percentage error map comparing both methods, and a line profile. Overall good visual agreement is confirmed in these images.

Fig. 3

Actual images of tissue mimicking phantoms. Absorption and reduced scattering images of four phantoms are shown over a range of optical properties. Results are shown for both wavelengths using standard SFDI as a reference and spatiotemporal modulation of light. In addition, relative error maps are shown and line profiles compared (dashed blue: standard SFDI, solid red: spatiotemporal modulation of light).

3.2.

In Vivo Experiments

Images from the still hand are shown in Fig. 4 for both wavelengths, in absorption and reduced scattering. Results from standard SFDI are shown, as well as spatiotemporal modulation of light, a percentage error map comparing both methods, and a line profile. Overall good visual agreement is confirmed in these images. Mean errors were calculated for all the pixels in a $50\times 50\mathrm{m}{\mathrm{m}}^2$ region on the back of the hand. For 665 nm, the mean error is $1.9\%\pm 2.0\%$ in absorption and $2.0\%\pm 0.7\%$ in reduced scattering. For 860 nm, the mean error is $3.4\%\pm 1.9\%$ in absorption and $0.3\%\pm 1.7\%$ in reduced scattering. These results confirm the phantom experiments with similar error values on average and allow to appreciate the image quality of the extracted optical properties maps. In particular, the vein structures are clearly visible on the absorption images confirming the potential for the spatiotemporal modulation of light method to perform adequately for clinically relevant use.

Fig. 4

Actual images of a still hand. Absorption and reduced scattering images of a still hand are shown over a range of optical properties. Results are shown for both wavelengths using standard SFDI as a reference and spatiotemporal modulation of light. In addition, relative error maps are shown and line profiles compared (dashed blue: standard SFDI, solid red: spatiotemporal modulation of light).

Finally, results from the arm-cuff occlusion experiment are shown in Fig 5. Significant decrease in oxygen saturation is visible during the occlusion, with values going from 78% down to 49%, following by a hyperemia phase after the release of the occlusion with a slow decrease back to the resting state.

Fig. 5

Arm-cuff occlusion results. An arm-cuff occlusion was performed for 4 min, following 3 min of resting state, and followed by 3 min of recovery. Images were acquired using spatiotemporal modulation of light at 100 fps and processed for oxygenation. Top images show oxygen saturation at three time points during the baseline resting measurement, the occlusion, and after the release. The saturation of a point in the hand (blue square) is shown in the plot below. Time points of the above images are indicated by arrows.

A movie is shown in Fig. 6. Note that the movie frame rate has been decreased from 100 to 25 fps to reduce the size of the file.

Fig. 6

Video of the arm-cuff occlusion (Video 1, MP4, 74.3 MB [URL: https://doi.org/10.1117/1.JBO.24.7.071610.1]).

4.

Discussion

In this work, we present the application of a method, spatiotemporal modulation of light, to perform real-time (100 fps), wide-field ($13\times 13\mathrm{c}{\mathrm{m}}^2$), and quantitative imaging of blood parameters in vivo. This method relies on the modulation of light sources in time to encode wavelengths and on the modulation of the projection of these sources in space to extract the optical properties of living tissues. Combined together, these spatiotemporal modulations allow to gather multispectral quantitative optical information in real time (here at 100 fps and postprocessed). Altogether the method enables to provide healthcare professionals with interpretable information that can be used in real time to guide decision making in challenging clinical situations, such as surgery.

Compared to previous work, this study augments the capacity for performing multispectral diffuse optical imaging quantitatively at a high frame rate and over a large field-of-view. For instance, sequential acquisition of two wavelengths using both SSOP and laser speckle imaging acquisition methods led to 16 fps imaging of simultaneous blood parameters and perfusion imaging over a wide field-of-view ($15\times 15\mathrm{c}{\mathrm{m}}^2$).¹² In more recent work, compressed sensing was used along with square patterns projections and phase shifting leading to 12.5 fps imaging of blood parameters over a smaller field-of-view ($3.5\times 3.5\mathrm{c}{\mathrm{m}}^2$).¹³

However, this work comes with limitations. First, in this implementation, the SSOP method results in slight image degradation ^9–11 and the 3-D profile of the sample is not measured, which can cause errors in extracting optical properties onto complex, non-flat objects ($\sim 10\%$ error per cm for absorption).²⁶ Second, noise in oxygenation data, mainly caused by the poor quantum efficiency of the camera at 860 nm, as well as slight synchronization errors between the camera and the sources, resulted in the impossibility to observe fine variations such as the pulsatile nature of blood flow. Improvements in the imaging system will include stronger illumination at 860 nm and a custom light source for optimal camera-source synchronization to allow the observation of finer variations of oxygenation. Third, currently images are acquired in real time but postprocessed following acquisition. We are currently working on a hardware and software solutions enabling both real-time acquisition and real-time processing, making this technology deployable for real-time visualization of the surgical field.^24,27,28 Finally, while the concept can be expanded to virtually many more wavelengths, there is a significant instrumental challenges in acquiring properly all wavelengths with a good enough SNR, and within the limitations of the spectral quantum efficiency and the dynamic range of the camera. For instance, in this work, the 860-nm optical properties, necessary to optimally separate oxy- and deoxyhemoglobin, shows significantly more noise compared with 665 nm due to the decrease of the spectral quantum efficiency of silicon-based camera technology. While manageable for a few wavelengths, this certainly limits the acquisition of a large number of wavelengths concurrently.

One advantage of this method though is its capacity to work in real time. Current CMOS technology offers cameras at a very reasonable price permitting good resolution and dynamic range imaging up to 500 fps. Our GP-GPU implementation of the temporal DFT in CUDA permits processing down to $18\mu \mathrm{s}$ per wavelength at a resolution of $512\times 512\mathrm{pixels}$.²⁰ However, the rolling window processing approach comes at the cost of a short delay between the acquired scene and the displayed information (here $1/10\mathrm{th}$ of a second, i.e., 100 ms). One potential solution to decrease this delay is to increase the acquisition rate (500 fps for instance) but it requires to increase the light fluence from the sources to maintain proper SNR and prevent noise in the images.

Finally, current state-of-the-art processing methods in the spatial frequency domain include optimized LUTs (down to 18 ms for $572\times 672\text{pixels}$),²⁴ as well as deep learning and machine learning methods (down to 200 ms for $696\times 520\text{pixels}$),^28,29 and soon, faster methods that will enable real-time processing and visualization of data will certainly be developed. Efforts are ongoing to combine rapid processing of SSOP data, with spatiotemporal modulation of light, onto a clinically compatible device to test and validate the capacity for diffuse optical imaging to provide relevant data in real time during concrete clinical applications.

5.

Conclusion

In this work, we presented an approach for real-time, wide-field, and quantitative imaging of physiologically and clinically relevant data onto living tissues. Spatiotemporal modulation of light enables true real-time quantitative multispectral imaging opening unprecedented possibilities for translating such technology to clinically relevant applications where time is constrained, such as surgery. This study contributes to the translation of diffuse optical imaging methods to relevant clinical applications.