1.

Introduction

Tissue oxygenation (${\mathrm{StO}}_2$) is a measure of the amount of oxygen in biological tissue and is often estimated by computing the fraction of oxygenated hemoglobin over total hemoglobin. ${\mathrm{StO}}_2$ is a useful clinical biomarker for tissue viability, the continuous monitoring of which is valuable for surgical guidance and patient management.^1,2 Abnormal levels of ${\mathrm{StO}}_2$ are indicative of many pathological conditions, such as sepsis, diabetes, and chronic obstructive pulmonary disease.^3–5

One of the most widely used techniques for measuring physiological oxygen levels is pulse oximetry. Despite its ubiquity, robustness, and low cost, pulse oximetry requires a pulsatile arterial signal and only provides a systemic measure of oxygenation.^6,7 The majority of existing devices for local assessment of ${\mathrm{StO}}_2$ are based on near-infrared (NIR) spectroscopy. NIR spectroscopy quantifies the concentrations of oxygenated and de-oxygenated hemoglobin by characterizing tissue absorption of light at wavelengths typically between 650 and 1000 nm.⁸ Similar to pulse oximetry, spectroscopic probes require direct contact with tissue. These measurements can be noisy as they are sensitive to pressure and sample movement.^9–11 Compared with tissue probes, spectroscopic imaging techniques are advantageous as they provide non-contact readings of oxygen saturation at a high spatial resolution and large field of view.¹² Nevertheless, continuous-wave NIR spectroscopy assumes constant scattering, which could be a source of error as scattering coefficients are often spatially non-uniform. Therefore, to accurately determine oxygen saturation, it is imperative to separate the effects of optical properties, including absorption (${\mu }_a$) and reduced scattering coefficients (${\mu }_s^{\prime }$). Spectrally constrained reconstructions have been shown to be useful in measuring chromophore concentrations and ${\mu }_s^{\prime }$, but this technique tends to require complex instrumentation and suffers from limited fields of view.¹³

In recent years, spatial frequency-domain imaging (SFDI) has emerged as a promising technique for measuring tissue optical properties. SFDI quantifies optical properties by projecting structured-light and characterizing the modulation transfer function of tissues in the spatial frequency domain.¹⁴ Oxygenation can subsequently be determined by fitting chromophore concentrations to the measured absorption coefficients using the Beer–Lambert law. In addition to isolating the effect of tissue scattering, SFDI is a wide-field, non-contact technique that can be implemented using a simple setup that includes a camera and a projector. These advantages make SFDI suitable for many clinical applications that necessitate accurate ${\mathrm{StO}}_2$ measurements, such as burn wound assessment,^15,16 pressure ulcer staging and risk stratification,¹⁷ image-guided surgery,^7,11 and cancer therapy evaluation.¹⁸

Despite its growing use in various applications, there are several factors that limit the clinical translation of SFDI. First, compared with probe-based oximetry, SFDI components are costly. For example, digital micromirror devices or spatial light modulators are often used to produce programmable structured illumination. Second, SFDI requires carefully controlled imaging geometries, which can be difficult to achieve in clinical settings. Moreover, conventional SFDI requires six images per wavelength (0, $\frac{2}{3}\pi$, and $\frac{4}{3}\pi$ phase offsets at two spatial frequencies) and a pixel-wise lookup table (LUT) search to generate a single optical property map. For robust oxygenation estimates, absorption coefficients at a minimum of two wavelengths are needed, and an additional least square fitting step is performed [Fig. 1(a)].¹⁹ Previous work has shown that real-time imaging can be achieved with single-snapshot acquisition²⁰ and either an optimized LUT²¹ or a machine learning inversion method.²² However, single image acquisition and frequency filtering often result in image artifacts and high per-pixel error.²³ Therefore, wide-field, rapid, and accurate ${\mathrm{StO}}_2$ measurement still remains a challenge.

Fig. 1

Comparison of (a) SFDI, (b) GANPOP, and (c) OxyGAN ${\mathrm{StO}}_2$ techniques. Ground truth SFDI uses 24 input images (2 spatial frequencies, 3 phases, and 4 wavelengths), while GANPOP and OxyGAN use 2 input images (1 spatial frequency, 1 phase, and 2 wavelengths). SFDI and GANPOP absorption maps are subsequently fit to basis chromophores to estimate ${\mathrm{StO}}_2$. OxyGAN directly calculates oxygen saturation with a single network, reducing compounding errors and processing time.

In recent years, convolutional neural networks (CNNs) have emerged as a powerful tool in many medical imaging-related tasks.^24,25 By employing convolutional filters followed by dimension reduction and rectification, CNNs are capable of extracting high-level features and interpreting spatial structures of input images.²⁶ For image translation tasks, generative adversarial networks (GANs) improve upon conventional CNNs by utilizing both a generator and a discriminator²⁷ to effectively model a complex loss function. The two components are trained simultaneously, with the generator learning to produce realistic output and the discriminator to classify the generator output as real or fake. Recently, GANs have been employed to predict optical properties from single structured-illumination images (GANPOP).²⁸ As a content-aware network, this technique significantly improves upon the accuracy of model-based single-snapshot techniques in estimating optical properties. However, to compute ${\mathrm{StO}}_2$ with the GANPOP approach, multiple wavelength-specific networks must be run to first estimate absorption coefficients, followed by chromophore fitting, which compounds errors and increases computational demand [Fig. 1(b)]. In this study, we present an end-to-end technique for computing ${\mathrm{StO}}_2$ directly from structured-illumination images using GANs (OxyGAN). OxyGAN maps ${\mathrm{StO}}_2$ from single-snapshot images from 659- and 851-nm sinusoidal illumination. We train generative networks to estimate both uncorrected and profile-corrected ${\mathrm{StO}}_2$ and compare the performance of the end-to-end architecture versus intermediately calculating optical properties. We accelerate OxyGAN model inference by importing the framework into NVIDIA TensorRT for efficient deployment. Finally, we demonstrate real-time OxyGAN by recording its estimation over the course of a 3-min occlusion experiment.

2.

Methods

For training and testing of OxyGAN, single structured-illumination images were acquired at two different wavelengths (659 and 851 nm) and paired with registered oxygenation maps. Ground truth oxygenation is obtained using the absorption coefficients measured by conventional SFDI at four wavelengths (659, 691, 731, and 851 nm). Experiments were conducted using both profile-corrected^29,30 and uncorrected ground truth. The training set included human ex vivo esophagus samples and in vivo feet. OxyGAN was evaluated using unseen tissues of the same type as the training data (human in vivo feet) and in different tissue types (human in vivo hands and a pig in vivo colon). Its performance was additionally compared with single-snapshot optical properties (SSOP)^23,31 as a model-based benchmark that utilizes a single structured-light image.

2.3.

OxyGAN Framework

In this study, we pose ${\mathrm{StO}}_2$ mapping as an image-to-image translation task. OxyGAN uses an adversarial training framework to accomplish this task (Fig. 2). Specifically, OxyGAN is a conditional generative adversarial network (cGAN) that consists of two CNNs—a generator and a discriminator. Both networks are conditioned on the same input data, which are single structured-light images in our case. First proposed in Ref. 33, the cGAN structure has been shown to be an effective solution to a wide range of image-to-image translation problems.³⁴ While conventional single-network CNNs require simple, hand-crafted loss functions, cGANs can be more generalizable because the discriminator can effectively learn a complex loss function.

Fig. 2

The OxyGAN framework. OxyGAN produces ${\mathrm{StO}}_2$ maps directly from single-phase SFDI images with 659- and 851-nm illumination. The generator is a fusion network that combines the properties of a U-Net and a ResNet. The number under each block describes the number of channels. The discriminator is a PatchGAN classifier with a receptive field of $70\times 70\text{pixels}$. The discriminator trains to classify the current image pair versus an input–output pair randomly sampled from a pool of 64 previously generated images.

For the OxyGAN generator, we implement a fusion network that combines the properties of a U-Net and a ResNet (Fig. 2).^35,36 Similar to a U-Net, the OxyGAN generator is an encoder–decoder setup with long skip connections between the two branches on the same level. However, OxyGAN also includes short skip connections within each level and replaces U-Net concatenation with additions, making the network fully residual.^28,37 The residual blocks on each level consist of five $3\times 3$ convolutional layers, with residual additions between the outputs of the first and the fourth layers. Dimension reduction on the encoder side and expansion on the decoder side is achieved with $2\times 2$ maxpooling and $3\times 3$ up-convolutions, respectively. We use regular ReLUs for the encoder and leaky ReLUs with a slope of 0.2 for the decoder. A final $3\times 3$ convolution followed by a $Tanh$ activation function is applied to generate the output. The discriminator is a three-layer PatchGAN with leaky ReLUs ($\text{slope}=0.2$),³⁴ which results in a receptive field of $70\times 70\text{pixels}$. To stabilize the training process, we incorporated spectral normalization after each convolution layer in both the generator and the discriminator.³⁸ We use an adversarial loss of

Eq. (3)

$${\mathcal{L}}_{\mathrm{cGAN}}(G,D)={\mathbb{E}}_{x,y}[\log (D(x,y))]+{\mathbb{E}}_x[\log (1-D(x,G(x)))],$$

where $G$ is the generator ($G:X\to Y$) and $D$ is the discriminator.³⁴ During training, $G$ tries to minimize this objective while its adversary, $D$, tries to maximize it. The discriminator is trained to classify both the current input–ground truth pair and an image pair randomly sampled from a buffer of 64 previously generated images. Determine if a given pair of images forms a correct reconstruction for a given input. This classification is made from data that include the current input–ground truth pair and an image pair randomly sampled from a buffer of 64 previously generated pairs. Additionally, an $L_1$ loss is included to improve the generator performance and training stability:

Eq. (4)

$${\mathcal{L}}_{L1}(G)={\mathbb{E}}_{x,y}[{\Vert y-G(x)\Vert }_1].$$

The full objective function is expressed as

Eq. (5)

$$\underset{G}{\min }\underset{D}{\max }\mathcal{L}(G,D)={\mathcal{L}}_{\mathrm{cGAN}}(G,D)+\lambda {\mathcal{L}}_{L1}(G),$$

where $\lambda$ is a hyperparameter that controls the weight of the $L_1$ loss term and was set to 60. OxyGAN models solved this objective using an “Adam” solver with a batch size of 1.³⁹ $G$ and $D$ weights were both initialized from a Gaussian distribution with a mean and standard deviation of 0 and 0.02, respectively. These models were trained for 200 epochs, and a constant learning rate of 0.0002 was used for the first 100 epochs. The learning rate was linearly decreased to 0 for the second half of the training process. The full algorithm was implemented using Pytorch 1.0 on Ubuntu 16.04 with a single NVIDIA Tesla P100 GPU on Google Cloud.²⁸

2.4.

Data Split and Augmentation

In this study, we conducted separate experiments to estimate both uncorrected (${\mathrm{StO}}_2$) and profile-corrected oxygenation (${\mathrm{StO}}_{2,\mathrm{corr}}$) from the same single-snapshot structured-light image input. These networks were trained and tested on $256\times 256\text{-}\text{pixel}$ patches paired with registered oxygenation maps. To generate training datasets, each $520\times 696$ image was segmented at a random stride size, which resulted in $\sim 30$ image pairs per sample. The input data were arranged in a way so that the three image channels normally used for color were efficiently utilized (Fig. 3). The first and second channels are flat-field corrected, single-phase illumination images at 659 and 851 nm, respectively. To account for system drift over time, we included the ratio between demodulated AC ($M_{\mathrm{AC}}$) and DC magnitude ($M_{\mathrm{DC}}$) of the reference phantom in the third channel. Reference measurements were taken on the same day as the tissue measurements, in the same way as conventional SFDI workflows. As shown in Fig. 3, the ratios at 659 and 851 nm alternate in a checkerboard pattern to account for any spatial variations.

Fig. 3

Images of the three input channels. Channels 1 and 2 are flat-field corrected single-snapshot SFDI images at 659 and 851 nm. Channel 3 gives ratios between demodulated AC ($M_{\mathrm{AC},\hspace{0.17em}\mathrm{ref}}$) and DC magnitude ($M_{\mathrm{DC},\hspace{0.17em}\mathrm{ref}}$) of the reference phantom. The ratios at 659 and 851 nm are alternated to form a checkerboard pattern, as shown in the $4\times 4\text{-}\text{pixel}$ template on the right.

To prevent overfitting of the models, we augmented the training data by flipping the images horizontally or vertically. During each epoch, both flipping operations occurred with a 50% chance and were independent of each other. Data augmentation was important for this study because of the small size of the training set and because the testing set includes new object types never seen in training. Additionally, since the classification task of the discriminator was easier than the generator, we applied the one-sided label smoothing technique when training the discriminator. In short, the positive (real) targets with a value of 1 were replaced with a smoothed value (0.9 in our case). This was implemented to prevent the discriminator from becoming overconfident and using only a small set of features when classifying output.⁴⁰

3.

Results

The average NMAEs are reported in Table 1 for SSOP, GANPOP, and OxyGAN tested on human feet, hands, and in vivo pig colon. The hands and feet are from different healthy volunteers with a wide range of pigmentation levels (Fitzpatrick skin types 1 to 5). It is worth emphasizing that the in vivo hands and pig colon were completely new tissue types that were not represented in the training set. On average, OxyGAN outperforms SSOP and GANPOP in accuracy by 24.04% and 6.88%, respectively, compared with uncorrected SFDI ground truth. Compared with profile-corrected ground truth, the improvement of OxyGAN over SSOP and GANPOP becomes 24.89% and 24.76%, respectively.

Table 1

NMAE of StO2 predicted by SSOP, GANPOP, and OxyGAN on both uncorrected and profile-corrected SFDI ground truth.

Figure 4 compares the results of profile-corrected SFDI, SSOP, and OxyGAN applied to a sample of each testing tissue type. Lower errors and fewer image artifacts are observed in OxyGAN results. Error plots highlight the fringe artifacts commonly observed parallel to the illumination patterns in SSOP. As expected, both SSOP and OxyGAN exhibited higher errors in the pig colon, which had more complex surface topography and made single-snapshot predictions more difficult.

Fig. 4

Comparison of profile-corrected SFDI, SSOP, and OxyGAN ${\mathrm{StO}}_2$ results: (a) in vivo human foot, (b) in vivo human hand, and (c) in vivo pig colon.

We additionally implemented OxyGAN on a video of a volunteer’s hand during an occlusion study (Fig. 5). The average oxygen saturation was calculated for a region of interest highlighted by the red box in Fig. 5(c) and was compared with the SFDI ground truth in Fig. 5(d). OxyGAN accurately measures a large range of oxygenation values and shows strong and stable agreement with conventional SFDI.

Fig. 5

Video of the occlusion-release experiment: (a) input at 659 nm, (b) input at 851 nm, (c) ${\mathrm{StO}}_2$ map predicted by OxyGAN, and (d) ${\mathrm{StO}}_2$ trend measured by OxyGAN and ground truth SFDI over time (Video 1, MP4, 3.7 MB [URL: https://doi.org/10.1117/1.JBO.25.11.112907.1]).

4.

Discussion

In this study, we have described a fast and accurate technique for estimating wide-field tissue oxygenation from single-snapshot structured-illumination images using GANs. As shown in Table 1, OxyGAN accurately measures oxygenation not only for sample types represented in the training set (human feet) but also for unseen sample types (human hands and pig colon). This supports the possibility that OxyGAN can be robust and generalizable. The occlusion video (Fig. 5) further demonstrates the ability of OxyGAN to accurately measure a wide range of tissue oxygenation levels and to detect changes over time.

Compared with training separate GANPOP networks to first estimate absorption coefficients, OxyGAN produces an average improved accuracy of 15.8%. Moreover, a greater improvement is observed in profile-corrected experiments. One potential explanation for this is that the errors in absorption coefficients due to uncertainties in profilometry estimation propagate and result in a larger error in oxygenation measurements. Additionally, compared with separate GANPOP models, the end-to-end OxyGAN approach requires only one network and bypasses the Beer–Lambert fitting step, thus greatly reducing the computational cost for training and inference. For example, training a network on 350 patches took $\sim 2.2\mathrm{h}$ or 40 s per epoch on an NVIDIA Tesla P100 GPU. Training separate GANPOP networks would take double the amount of time and memory. To achieve real-time ${\mathrm{StO}}_2$ mapping, we first converted the trained model to the Open Neural Network Exchange (ONNX) format. We then imported the ONNX model into NVIDIA TensorRT 7 for reduced latency and optimized inference. For testing, OxyGAN inference on a Tesla P100 took $\sim 0.04\mathrm{s}$ to generate a $512\times 512$ oxygenation map. This is 8 times faster than computing optical properties with two GANPOP networks and $\sim 10$ times faster than two GANPOP inferences followed by a Beer–Lambert fitting step. We expect OxyGAN to process $1024\times 1024$ images at a similar framerate (25 Hz) on a quad-GPU workstation.

To evaluate model performance, we benchmarked OxyGAN by comparing it with a single-snapshot technique based on a physical model (SSOP). Table 1 shows that, in estimating both uncorrected and profile-corrected oxygenation, OxyGAN achieves higher accuracy than SSOP in all tissue categories. In addition to improved average accuracy, OxyGAN results also contain fewer subjective image artifacts (Fig. 4). These benefits are more pronounced for samples with complex surface topography, such as the pig gastrointestinal sample. Unlike SSOP, which relies on Fourier domain filtering, OxyGAN utilizes both local and high-level features. As a content-aware, data-driven approach, OxyGAN has the potential to learn the underlying distribution of the data and accurately infer oxygenation in regions with low signal or non-uniform surface structures.

In Table 1, we observe that GANPOP achieves similar accuracy to SSOP for profile-corrected ground truth. This is expected for several reasons. First, the training set used in this study is smaller than in the original GANPOP paper,²⁸ excluding in vivo hands and tissue-mimicking phantoms. Second, for physical model-based techniques, such as SSOP, the optical property errors due to surface topography variation are correlated across wavelengths and can later be reduced by chromophore fitting. For instance, for surface normal vectors pointing further away from the detector, the predicted absorption coefficients will be overestimated for both 659 and 851 nm. However, the fitting of hemoglobin concentrations oxygen saturation, which relies on the ratios of absorption coefficients, may mask the intermediate optical property errors. Because the GANPOP networks are trained independently for 659 and 851 nm, the loss function does not learn these correlations, resulting in smaller improvements in accuracy over SSOP for ${\mathrm{StO}}_2$ measurements than for optical property measurements. This observation also provides some intuition for why the OxyGAN network might improve accuracy over GANPOP. Because OxyGAN is trained on multi-wavelength input and the loss function is computed from the ${\mathrm{StO}}_2$ estimate, it is capable of modeling correlations between absorption at different wavelengths and learning to reduce the effects of varying surface topography. Furthermore, a higher error rate is observed for the pig colon sample for both GANPOP and OxyGAN, likely due to this tissue type not being included in the training set and the complex topography of the colon specimen compared with other training samples.

The architecture of OxyGAN is based on the GANPOP framework.²⁸ The generator combines the features of both the U-Net and the ResNet, in that it incorporates both short and long skip connections and is fully residual. As discussed in Ref. 28, this fusion generator has advantages over most other existing architectures because it allows information flow both within and between levels, which is important for the task of optical property prediction. In this study, we empirically trained a model with a standard U-Net generator. The model performed well on sample types included in the training set; however, it collapsed and was unable to produce accurate results when tested on unseen sample types, such as human hands. Compared with GANPOP, OxyGAN employs data augmentation in the form of horizontal and vertical flipping, which is important for preventing overfitting of models trained on small datasets. OxyGAN also utilizes label smoothing in training the discriminator, which further improves model performance and overall training stability. Finally, we found that adding a channel of checkerboard reference phantom measurements to the 2-wavelength structured-light inputs improves accuracy for measurements taken on different days, allowing OxyGAN to take system drift into account similarly to conventional SFDI.

In the future, more work could be done to optimize the algorithm of OxyGAN to further improve the data processing speed. The model could be trained and tested on larger datasets that span a wider range of tissue types or scenarios that might be encountered clinically. To develop a robust and generalizable model, future work should train on data with a range of spatial frequencies acquired on several different instruments. Domain adaptation techniques could also be implemented on the trained models to improve robustness to different imaging geometries. In addition, similar to other single-snapshot techniques, one limit of OxyGAN is its expected sensitivity to ambient light. Moreover, oxygenation mapping using SFDI structured illumination is currently limited because it has a shallow depth of field and requires precisely controlled imaging geometry, making its clinical adoption particularly challenging. One alternative is to use random laser speckle patterns as structured illumination, which could be less costly than SFDI projection systems and more easily incorporated into endoscopic applications and may avoid fringe artifacts due to sinusoidal illumination.⁴² Monocular depth estimation could also be incorporated for profile-correction without requiring a projector and profilometry.^43,44 Furthermore, a more sophisticated LUT could be developed to directly estimate ${\mathrm{StO}}_2$ from SSOP data, which models the correlations between reflectance measurements at different wavelengths and the underlying tissue oxygenation. This pixel-wise estimation may provide a more accurate baseline that will help quantify the benefit of the content-aware aspect of OxyGAN. Finally, data-driven methods may be useful for taking higher-order optical property effects, such as the scattering phase function, into account.

5.

Conclusion

In this study, we have presented an end-to-end approach for wide-field tissue oxygenation mapping from single structured-illumination images using cGANs (OxyGAN). Compared with both uncorrected and profile-corrected SFDI ground truth, OxyGAN achieves a higher accuracy than model-based SSOP. It also demonstrates improved accuracy and faster computation than two GANPOP networks that first estimate optical absorption. This technique has the potential to be incorporated into many clinical applications for real-time, accurate tissue oxygenation measurements over a large field of view.