2.1.

HRME Image Acquisition and Pathology Interpretation

A HRME image dataset acquired by use of a low-cost, point-of-care HRME device that imaged the esophageal epithelium of patients undergoing endoscopy for ESCC screening¹⁸ was used in the study. The HRME images were obtained sequentially from patients enrolled in a clinical trial comparing standard of care Lugol’s chromoendoscopy (LCE) to LCE+HRME at three sites: First Hospital of Jilin University (Changchun, China), the Cancer Institute at The Chinese Academy of Medical Sciences (Beijing, China), and Baylor College of Medicine (Houston, Texas, United States) from December 2014 to November 2016, approved by the Institutional Review Board at Baylor College of Medicine [IRB# H-34973]. The details of image acquisition have been described in previous studies.^18,19

The HRME imaging system consisted of a compact epi-fluorescence fiber optic microscope, which provides $1000\times$ magnification views of epithelial tissue and subcellular features to distinguish cancerous from benign tissue after staining with a topical fluorescent dye, 0.01% proflavine hemisulfate. It has been evaluated in various anatomical sites, such as the cervix, anus, mouth, throat, and esophagus.^5,19 Due to its low cost and reusability, HRME can have a high impact in resource-limited global settings. The HRME principles and schematics have been described previously.¹⁸ Typically, the HRME device conducts imaging through a fused optical fiber bundle with a cross-sectional diameter of 800 and $4.4\mu \mathrm{m}$ lateral spatial resolution (an individual fiber strand diameter as little as $4\mu \mathrm{m}$). For a conventional microscopic probe, the input and the output at the distal and proximal ends of the fibers demonstrate the same pattern. The fiber strands are positioned as close to each other as possible, and the size of the fiber strands usually defined image resolution for a registering camera of high-resolution. The light illumination measures are collected from a continuous part of tissue and registered on a camera correspondingly. Figure 2(a) demonstrates the optical fiber pattern generated by the fiber strands of the fused optical fiber bundle.

Fig. 2

Schematics of image acquisition by means of (a) a conventional fused endoscopic fiber probe; (b) an end-expandable, unfused endoscopic fiber probe.

The density of the nuclei represented the metric for diagnosis and differentiation between neoplastic and non-neoplastic images. All image sites were biopsied, and biopsy histopathology results served as the gold standard. High-grade dysplasia and ESCC were classified as neoplastic; normal squamous epithelium, esophagitis, and low-grade dysplasia were classified as non-neoplastic.

All HRME images were standardized in size of $960\text{pixels}\times 1280\text{pixels}$. A Gaussian filter with standard deviation of 2 pixels was applied to remove the comb pattern introduced by the sparse data of optical fiber bundles. A contrast-limited adaptive histogram equalization with a clip limit of 0.005 was employed to enhance the image contrast.

2.2.

Quality Control and Image Selection

In addition to IQ control conducted as described in the previous study,¹⁹ bioengineers further categorize the acquired HRME images into three types of perceptual image qualities: good, intermediate, and poor. Images with good quality should demonstrate an FOV where nuclei can be clearly visualized. Images with intermediate quality showed mild motion blur or defocus aberration that could affects approximately a quarter of the imaged area. Images with poor quality were classified by factors, such as severe motion blur, obstructed vision, or image corruption for half of the image area or more. Only images of good and intermediate perceptual quality were selected and used in this study.

2.3.

Simulated Image Data for End-Expandable Optical Fiber Probe

A virtual imaging trial was performed to simulate LR images that would be acquired using the proposed novel optical fiber probe by use of the acquired HRME images. As shown in Fig. 2(b), one can find a schematic of the concept to an end-expandable optical fiber probe. Unlike conventional image acquisition through an optical fiber bundle [Fig. 2(a)], the end-expandable optical fiber bundle can form a “brush” at the input end [Fig. 2(b)]. The input elements of such lightguide are fiber strands, which are placed at a distance from each other and all together they represent a sparse data set. Thus, the optical fiber “brush” collects fluorescent emission from a surface partially and discontinuously. To create LR images, we processed the simulated sparse image data acquired from the end-expandable endoscopic optical fiber probe [Fig. 2(b)] by filling in non-populated pixels of the sparse images [Fig. 3(b)]. Diagrams that depict the conventional and sparse image data acquisition and processing are shown in Fig. 3.

Fig. 3

(a) 1D-schematic of the conventional image data acquisition leading to the HR images; (b) the sparse image data acquisition and reconstitution from the compressed data set into the LR images. Red elements represent a fiber strand collecting light illuminated by the tissue surface and green elements are a set of pixels displayed as average value of light intensity in the adjacent fiber. (c) 2D-illustration of the simulated sparse image data; $s$ and $m$ are the length of side of FOV and ROI, respectively; $d_x$ and $d_y$ are the offsets in $x$ and $y$ directions, respectively.

To emulate the LR images that would be acquired from the novel end-expandable optical fiber probe, a degradation model that incorporates the fiber knock-out downsampling was applied to the HRME images. Assuming a single fiber strand with a diameter of $4\mu \mathrm{m}$, we considered a pixel size of $2\mu \mathrm{m}$ in this study. As shown in Fig. 3, given a fiber strand represented by a range-of-interest (ROI) block of $m\text{pixels}\times m\text{pixels}$ that resides in the center of an FOV block of $s\text{pixels}\times s\text{pixels}$, a random deformation offset with maximum of $d_x$ pixels and $d_y$ pixels in horizontal and vertical direction was considered, respectively. This was designed to depict the off-center deviation when defining the ROI block. The degradation model calculated the mean value over pixels within the ROI block and filled the pixels in FOV block with the derived value. In this study, three different parameters of the degradation model were investigated to evaluate the DL-SR performance: the fiber diameter $m$, the inter-fiber distance $s$, and the deformation offset $d$. The schematic of the proposed degradation model is shown in Fig. 4.

Fig. 4

Illustration of degradation models that incorporated various optical fiber probe parameters including (a) offset, (b) inter-fiber distance, and (c) fiber diameter.

Figure 5 shows examples of an HRME image, an intermediate sparse image and the LR image simulated by use of the degradation model, respectively. Here, the fiber diameter $m=6\mu \mathrm{m}$, the inter-fiber distance $s=12\mu \mathrm{m}$, and the deformation offset $d=1$.

Fig. 5

Example of an original HRME image, the same imaging area as sparse data captured by the proposed end-expandable optical fiber probe, and the LR image restored from the sparse data.

2.4.

Deep-Learning-Based Image Super-Resolution

Given an LR image ${\mathbf{I}}_{\mathrm{LR}}$ virtually acquired with the end-expandable optical probe, image super-resolution methods seek to produce an SR image ${\mathbf{I}}_{\mathrm{SR}}$ as an estimate of a HRME image that would be obtained from a conventional fused endoscopic fiber probe. However, this is a challenging ill-posed inverse problem. In recent years, DL-SR methods have been widely applied in various applications.^{13–15,20–24} Here, the well-studied super-resolution convolutional neural network (SRCNN)¹³ was employed to investigate the feasibility of DL-SR methods to improve quality of simulated LR images from the novel optical probe. Such analysis can be readily repeated with other more recent DL-SR approaches.^14,25 The SRCNN seeks to establish a mapping from the space of simulated LR images to the space of HR images

Fig. 6

Schematic of the SRCNN architecture used in the study.

The training and validation data for the SRCNN consisted of 206 and 50 paired HRME and corresponding simulated LR images, respectively. During training and validation, each image was randomly cropped into 10 patches with a size of $512\times 512$. In the testing stage, the full-size simulated LR images were used. The SRCNN was trained with Adam optimizer²⁶ with a learning rate of 0.0001. The network was trained for 300 epochs with a batch size of 8, and the model with best validation performance was used for downstream task evaluation. For various degradation parameters mentioned in Sec. 2.3, the SRCNN was retrained and evaluated. SRCNN were implemented with TensorFlow 2.0 and trained on NVIDIA GPUs.

2.5.

Image Quality Assessment and Statistical Analysis

To assess the DL-SR performance with consideration of various degradation parameters used to simulate the LR images, traditional IQ metrics, such as peak-signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM) were computed on the simulated LR and SR images. In addition, a binary detection task to determine whether esophageal neoplasia is present or not was performed by endoscopists on both SR images obtained from SRCNN and the original HRME ones. The accuracy and confidence level were evaluated to assess the task-based performance of the employed DL-SR method.

A total of four endoscopists (three experts, one novice) underwent standardized training in HRME image interpretation. Expert endoscopists were defined as having previously performed $>50$ HRME cases, whereas novices were new to the technology. All endoscopists viewed a set of training slides that demonstrated the features of neoplastic and non-neoplastic classification of HRME images, including nuclear size, crowding, and pleomorphism. All endoscopists were asked to interpret a series of original HRME images and generated SR images as neoplastic (high-grade dysplasia, ESCC) or non-neoplastic (normal squamous, esophagitis, low-grade dysplasia) along with their confidence level in their interpretation (high or low).

Assuming 80% power, $\alpha =0.05$, and two-sided test to establish equivalence between HR and SR images with equivalence limit of 0.15, a sample size of 120 images was calculated using sample-based variance estimates without continuity correction for binary data. The sensitivity, specificity, and accuracy of HRME and SR image interpretation of individual endoscopists was compared with histopathology as the gold standard. Unpaired t-test was computed to assess for differences in the sensitivity, specificity, and accuracy in endoscopists’ interpretation of HR and generated SR images.

3.1.

Performance Evaluation of DL-SR to Super-Resolve Simulated LR Images of End-Expandable Optical Fiber Probe

The DL-SR model was first trained and tested on HRME images and corresponding LR images simulated by use of a degradation model with fiber diameter $m=4\mu \mathrm{m}$, inter-fiber distance $s=8\mu \mathrm{m}$ and no offset imposed. Figure 7 shows two examples of the test images where perceptual IQ of the simulated LR images appeared to be improved by DL-SR. Figure 8 shows a zoomed-in, cutoff image patch of the original HR, simulated LR, and generated SR images and the corresponding cross-sectional profiles of optical intensity. Note that, as expected, the line profiles of the SR image aligned closer to the HR one, compared to the line profiles of the simulated LR image. Visual inspection confirms that details of subcellular features could be restored by use of SRCNN to obtain improved perceptual IQ.

Fig. 7

Examples of original HRME image, simulated LR image and corresponding SR image (top row) generated by the DL-SR method with magnification (bottom row).

Fig. 8

Cross-sectional line profile on selected images allows comparison of profiles of optical intensity associated with stained nuclei on original HR images, simulated LR images, and reconstructed SR images.

3.2.

Impact of Offset, Inter-Fiber Distances and Fiber Diameter on Traditional Image Quality Metrics

In this study, the impact of different parameters considered in the degradation model on traditional IQ metrics was evaluated. The offset $d$ was varied from 0 to $10\mu \mathrm{m}$, and the inter-fiber distance $s$ was changed ranging between 4 and $24\mu \mathrm{m}$. A fiber strand diameter $m$ ranging from 4 to $12\mu \mathrm{m}$ was applied. All SRCNN models were trained on paired of HRME and LR images simulated with various degradation parameter values. Traditional measures of IQ were assessed by computing PSNR and SSIM values on a test set consisting of 300 images, and these quantities are plotted on Fig. 9. In most cases, the SR images generated by the SRCNN demonstrated improvements across various offsets, inter-fiber distances and fiber diameters compared with their LR counterparts in terms of the traditional IQ metrics. Moreover, various degradation parameters showed different levels of impacts on the traditional IQ metrics of simulated LR and SR images. When increasing the offset of fibers, traditional IQ metrics of both LR and SR images decreased. Similar phenomena were observed when the inter-fiber distance and fiber diameters were increased. This is due to the more severe degradation model incorporated and thus DL-SR performs worse on images largely missing information when the network capacity or the number of training images were limited.

Fig. 9

Traditional IQ metrics show degradation of LR and limitation of improvement of SR images with reference to HR images: PSNR and SSIM values of LR (red) and SR (blue). The gray dashed line denotes a SSIM value of 0.95.

3.3.

Endoscopists’ HRME Image Interpretation of DL-SR Images

Among the 120 paired HRME images and SR images included in the study, 42 images (35%) were non-neoplastic and 78 images (65%) were neoplastic. The diagnostic performance of endoscopists on HR and SR images is demonstrated in Figs. 10 and 11 for the summation of all four endoscopists’ reads. Overall, the task-based IQ of SR images was comparable to that of HR images when endoscopists interpreted them in a post-hoc setting. On the original HR images, the endoscopists achieved an overall accuracy of 70.2% (standard deviation [SD], 5.4%), a sensitivity of 73.7% (SD, 14.1%), and a specificity of 63.7% (SD, 12.8%). On the SR images, the endoscopists did not significantly change their accuracy ($p=0.37$), sensitivity ($p=0.70$), or specificity ($p=0.07$).

Fig. 10

Diagnostic performance of endoscopists’ reading on HR and SR microendoscopy images shown in box and whisker plot. The cross and horizontal line in the box represents mean and median values of the endoscopists’ response. The bottom and the top end from the whiskers are the minimum and maximum values, respectively.

Fig. 11

(a) Sensitivity and specificity plot of individual endoscopist diagnosis on HR and SR microendoscopic images with high and low confidence, respectively. (b) Confusion matrix of diagnostic performance of readers on HR and SR images with low and high confidence, respectively. TP, true positive; FP, false positive; FN, false negative; and TN, true negative.

Overall, endoscopists had high confidence in their image interpretation in a mean 51.1% (SD, 7.7%) of HR images and 48.5% of SR images (SD, 9.0%, $p=0.68$). Among high confidence images, endoscopists had no significant difference in their accuracy (76.0% versus 74.8%, $p=0.73$), sensitivity (84.6% versus 88.3%, $p=0.45$), or specificity (45.9% versus 30.2%, $p=0.19$) on SR images compared to HR counterparts. Similarly, for low-confidence images, endoscopists did not have a significant difference in their accuracy (64.2% versus 60.1%, $p=0.39$), sensitivity (58.1% to 63.7%, $p=0.68$), or specificity (73.3% versus 55.8%, $p=0.14$) on SR images compared to HR images.