1.

Introduction

Colorectal cancer (CRC) is currently the third leading cause of cancer death in the United States, despite increased screening rates and a concomitant decline in incidence and mortality.^1,2 CRC commonly arises from adenomatous polyps, some of which may progress into invasive adenocarcinomas over time.³ This progression is thought to be slow and is dependent on the size and histology of the polyp.⁴ Screening for colorectal polyps via conventional white light colonoscopy has had a dramatic effect on reducing the overall morbidity and mortality due to CRC, as visible polyps can be safely removed during this procedure.⁵ However, small lesions, sessile (i.e., flat) lesions, or regions of occult dysplasia can be occasionally missed. A broad range of alternative strategies have been recently employed to improve endoscopic screening methods, including autofluorescence endoscopy with real-time image processing, narrow band reflectance imaging, and combined multimodal methods.^6–8 While these approaches have demonstrated improved sensitivity for early detection of neoplasia, false positive rates remain elevated, limiting widespread adoption of these methods for screening purposes.⁹

High-resolution endoscopic imaging modalities have been widely explored to improve detection of these focal regions of early dysplasia in gastrointestinal epithelium.^10–13 Laser scanning confocal imaging methods employing novel miniaturized distal scanning mechanisms based on microelectromechanical systems have been demonstrated to yield excellent lateral and axial spatial resolution.^14–16 Clinical trials of a hybrid endoscope/miniaturized laser scanning confocal microscope system have demonstrated utility in diagnosing focal regions of dysplasia in the setting of Barrett’s esophagus and during colonoscopy.¹⁷ These systems offer real-time, histology-level information to be displayed to the clinician at the time of endoscopy, potentially greatly improving biopsy targeting to locations most likely to show evidence of dysplasia.^18,19 Nonscanning, wide-field fiber bundle microendoscopy methods have also been demonstrated to yield high-resolution image data from the superficial epithelium with promising diagnostic accuracy despite the limited field of view,²⁰ restricted by the biopsy port ($\sim 2\mathrm{mm}$) through which these devices are deployed in vivo. While certain technical challenges that remain before microendoscopy methods may be widely disseminated in clinical applications, the combination of improved wide-field endoscopic methods and high-resolution microendoscopy methods may greatly improve upon currently achieved accuracy.^20,21

Furthermore, although several clinical trials of a high-resolution fiber bundle microendoscopy device have demonstrated excellent interobserver agreement in interpretation and classification of fiber bundle microendoscopy images of colorectal polyps,²⁰ there is currently no widely accepted, quantitative image analysis criteria that can be used to objectively classify fluorescence image data of colorectal polyps without human intervention. Previously demonstrated interobserver agreement of microendoscopic image interpretation is dependent on highly trained staff and physicians. This need for high levels of training can be limiting to the dissemination of microendoscopic screening devices, and quantifiable image analysis criteria is an option to grant unspecialized practitioners the tools for consistent diagnosis at a reduced burden of training.

Quantitative image analysis applied in other microendoscopy studies include texture features of ovarian cancer confocal microendoscopy, cell density in mouse model sarcomas using wide-field structured illumination, and wavelet decomposition in color colonoscopic video frames.^22–24 These algorithms quantify a range of image features, such as texture (including spatial-frequency content), cell nuclei image segmentation and counting, and discrete wavelet transform and covariance.^22–24 There are numerous image features that could be extracted for quantification, depending on the data acquired by the imaging modality, but for the quantification of image features in fluorescence microendoscopy of nonspecific staining of bulk tissue, we chose image features that trained clinicians would typically consider in their analysis, namely crypt shape, size, and homogeneity. Pathologists who study en face histopathology of colorectal tissue for signs of dysplasia take into account image features such as elongated or branched crypts, clusters of crypts or crypts of abnormal size, irregular contours, and stratification of the nuclei surrounding the crypt. While pathologists commonly use nucleic patterns as a marker for abnormality, due to the length of the colorectal tract, it is of greater relevance during microendoscopic screening to use a fiber bundle with a larger field of view than to acquire images that resolve individual nuclei. Therefore, the algorithm we developed focuses on quantifying the larger structures of epithelial morphology such as crypt area and circularity. Additionally, the extent to which image morphology varies in clinically normal colorectal tissue in patients with different clinical histories, such as diagnosis, chemotherapy, radiation, prior surgeries, or other therapy, has not been described or quantified. There is a significant need for a quantitative benchmark for future detection of occult dysplasia in patients who have a range of diagnoses and/or prior treatment, and a need for generalized image analysis tools for advanced microendoscopy imaging modalities within the gastrointestinal tract.

Any quantitative image analysis algorithm must be able to exclude motion-blurred or other low-quality images; this is of particular concern in the demanding environment of an in vivo endoscopy procedure. Approaches such as microarray image processors have demonstrated that the inherent variability in signal-to-background ratios is correlated with image quality.²⁵ Low signal-to-background intensity ratios can affect the ability of an algorithm to accurately detect image features, and qualitative image exclusion via manual inspection may introduce human bias and additional error. While some microendoscopy images may be significantly distorted, the parameters for excluding low-quality images for machine detection by observation alone may vary in accuracy when compared to a quantitative threshold based on a standardized metric.

The objective of the study presented in this paper is to explore the quantification of microendoscopy images in the clinically normal appearing tissue surrounding grossly visible lesions in recently resected colorectal specimens, in order to extract key differences that may be used to classify benign epithelial tissue from dysplastic tissue in future computer-aided diagnostic algorithms. We present both a quantitative image quality control algorithm to exclude low-quality images, as well as an intensity-adaptive quantitative image feature extraction algorithm (QFEA) to segment and quantify crypt morphology in images of superficial colorectal epithelium acquired via fiber bundle microendoscopy. Both algorithms were first validated using an extensive training library constructed from image data acquired from fresh porcine colorectal epithelium, a tissue histologically similar to human colorectal epithelium. In addition to individual image acquisition, consecutive images were acquired for mosaicking, to demonstrate the ability to improve the inherently small field of view in fiber bundle microendoscopy images, one prominent disadvantage of microendoscopic systems, using image registration of consecutive fields of view.²⁶ All individual images, and several mosaics, collected from human specimens were then analyzed for quality and subsequently quantified using QFEA. Images were analyzed to segment crypts within the superficial epithelium, and a centroid map was created for identified crypts. From each of these maps, quantitative features were calculated, focused on crypt-specific properties such as crypt area and circularity. While these metrics were calculated for proflavine fluorescence fiber bundle microendoscopy, this method could be broadly applied over a range of high-resolution microendoscopy imaging modalities.

2.

Materials and Methods

2.2.

Porcine Tissue

Microendoscopic images of freshly resected colonic porcine tissue were acquired, in collaboration with Siloam Springs Processing (Siloam Springs, Arkansas), a local slaughterhouse. One hundred images were acquired from four healthy animals for training validation of both the quality control and quantitative feature extraction algorithms. These images were acquired from regions $\sim 5\mathrm{cm}$ in length, with an average of 30 images acquired per region. Porcine colorectal epithelium, histologically similar in structure and scale to human colorectal epithelium, has been previously used as a human analog in myriad imaging studies.^29,30 Figure 1 shows a comparison of normal porcine and normal human colorectal tissue using microendoscopic and histopathology images.

Fig. 1

Normal human versus normal porcine colorectal epithelial tissue. (a) and (b) Microendoscopy images; scale bar is $100\mu \mathrm{m}$. (c) and (d) Histopathology images; scale bar is $100\mu \mathrm{m}$. (a) and (c) Normal human colorectal epithelium. (b) and (d) Normal porcine colorectal epithelium.

2.4.

Microendoscopy Image Acquisition

Upon receipt of the freshly resected porcine tissue, the colon specimen was longitudinally sectioned and cleaned using cold saline. Microendoscopy images were acquired immediately following topical application of proflavine (0.01% w/v in $1\times$ phosphate buffered saline) with a cotton swab to epithelial surface, with occasional reapplication of dye after several minutes as needed. Individual images were acquired by manually placing the distal end of the fiber bundle in contact with the epithelial surface of the tissue; consecutive images were acquired by slowly dragging the distal end over the epithelial surface for subsequent processing into mosaics. Sequential image stacks were acquired over 10 to 20 s (covering $\sim 1$ to 2 cm of epithelium), and all images were acquired in the range of 100 to 150 ms exposure and 0 to 5 dB gain. At the conclusion of the imaging study, porcine tissue was placed in 10% formalin for fixation overnight, prior to paraffin embedding, sectioning, and hematoxylin and eosin staining.

For human subject imaging, the resected colorectal tissue was first longitudinally sectioned and then pinned onto a paraffin block. Additional pins were placed 2 cm apart at regions of interest, and images were acquired along a linear path between the pins. Microendoscopy images were acquired immediately following topical application of proflavine (0.01% w/v in $1\times$ phosphate buffered saline) with a cotton swab to epithelial surface, with occasional reapplication of dye after several minutes as needed. Individual images were acquired by manually placing the distal end of the fiber bundle in contact with the epithelial surface of the tissue; consecutive images were acquired by slowly dragging the distal end over the epithelial surface for subsequent processing into mosaics. Sequential image stacks were acquired over 10 to 20 s (covering $\sim 1$ to 2 cm of epithelium), and all images were acquired in the range of 100 to 150 ms exposure and 0 to 5 dB gain. For each specimen, typically one to two regions were imaged and sectioned for histopathology confirmation. After imaging, the study pathologist (K.L.) sectioned the tissue along the path marked by the pins, which facilitated the correlation of histopathology to microendoscopy (Fig. 2).

Fig. 2

Examples of tissue preparation, microendoscopic image, and histopathology. (a) Freshly resected gross tissue, pinned to paraffin block; white arrows point to pins delineating imaging location. Scale bar is $\sim 2\mathrm{cm}$. (b) Microendoscopic image of untreated normal colon epithelium. Scale bar is $100\mu \mathrm{m}$. (c) Histopathology image of untreated normal colon epithelium. Scale bar is $100\mu \mathrm{m}$. Images have enhanced contrast and brightness for publishing.

Two-centimeter sections of transversely sliced tissue were taken from imaging sites, and diagnoses provided by a clinically trained gastrointestinal pathologist (K.L.). Tissue was classified as clinically normal, normal with underlying invasive adenocarcinoma, invasive adenocarcinoma, or tubular adenoma at each of the imaged sites.

2.5.

Quantitative Image Quality Control

We developed a quantitative image quality control algorithm to exclude low-quality images prior to image quantification, based on crypt-specific signal-to-background ratio. This signal-to-background is based on the contrast between the signal intensity within each crypt and the surrounding region; the crypt should be darker than the DNA-containing cell nuclei around it. If the ratio between the intensity values in the surrounding region and the intensity values in the crypt was too low, the algorithm lost sensitivity in image segmentation and quantification (Fig. 3). The image segmentation and quantification algorithm was applied to images of both low and high quality (contrast); specifically, the sensitivity and positive predictive value (PPV) of crypt identification was calculated. Images with low contrast were excluded from pilot study quantification of crypt image features.

Fig. 3

Example of image quality based on signal-to-background contrast. Top row: microendoscopy image of human epithelium, with superimposed yellow stars marking centroids, and yellow box labelling the selected crypt; scale bar is 250 µm. Bottom row: surface maps of pixel intensity in region marked by yellow box (top row). Unlabeled axes are in pixels; 100 pixels are approximately 75 µm. The respective lowest intensity of each selection was subtracted from the image arrays, so that the lowest pixel value is zero without altering the differences in intensity. The same smoothing function applied to the line profiles for quality ratio (QR) calculation was applied to these image arrays.

Prior to calculating the contrast (signal-to-background) ratio, which we have termed the QR, all images were preprocessed to reduce common intrascene variation, which can result from, e.g., varying amounts of mucosa or variations in staining due to the topography of the tissue. The algorithm first masked the original images to restrict analysis and image segmentation to only the active area of the fiber bundle, and then contrast the enhanced images, whereby intensity values in the image are mapped to new values so that 1% of the image pixels are saturated at the lower and upper contrast limits (in this preprocessing step, 1% was saturated at zero intensity and 1% was saturated at 255), serving to standardize the images by utilizing the full range of the intensity values, before calculating the QR. Images were masked, excluding any values outside a circle 1000 pixels ($750\mu \mathrm{m}$) in diameter, ensuring only the uniformly illuminated region of the active area is analyzed. A microendoscopy image collected from a uniformly fluorescent agarose phantom (2% w/v agarose, mixed with 0.01% w/v proflavine, in distilled water) displayed in the full range of colors in the colormap demonstrates this process (Fig. 4).

Fig. 4

Example of unmasked microendoscopy image. Arrow points to unwanted light from outside the fiber bundle. The circle is 1000 pixels in diameter and represents the image area after masking. Image taken of agarose phantom mixed with 0.01% (w/v) proflavine. Scale bar is $250\mu \mathrm{m}$.

To calculate the QR, the image quality control algorithm extracted vertical and horizontal line profiles (90 total) from the image, at intervals of 40 pixels (Fig. 5). This was done to ensure that line profiles would cross the majority of crypts at least once, accounting for the physical scale of crypts (ranging 75 to $250\mu \mathrm{m}$ in diameter) and the scale of the pixels at the image plane (Fig. 5). Each line profile was smoothed to eliminate small intensity variations due to the fiber bundle pattern itself. This fiber pattern exhibits a spatial frequency of $\sim 10\text{pixels}$; smoothing eliminates this periodic variation [Figs. 6(a)–6(d)]. Then, local regional minima and maxima were determined for each line profile, and the ratio between each minimum and its adjacent maximum was calculated [Fig. 6(e)]. The regional minima and maxima were determined by comparing regions of eight values (eight-connected neighborhoods) to surrounding values and outputting a binary map of the values of the local minima or maxima. All ratios within each image were then averaged to output a single global average QR metric for the image. Figure 7 shows the flow diagram of steps for preprocessing (image masking and contrast enhancement) and QR calculation.

Fig. 5

Example of typical crypt sizes and chosen grid line spacing. (a) Microendoscopic image of healthy porcine colorectal tissue. Inset shows a normal crypt with a width of 80 pixels. Vertical yellow lines represent the grid lines, with spacing of 40 pixels (yellow double-headed arrow), that are used to calculate QR in the image quality control algorithm. Scale bar is $250\mu \mathrm{m}$. (b) Example of grid structure and spacing used to extract line profiles for QR calculation.

Fig. 6

Smoothing and QR calculation. (a) and (b) Microendoscopy images of flat-field illumination showing individual fiber artifacts, before and after smoothing is applied. Axes display distance in pixels; scale bar is $6\mu \mathrm{m}$. (a) Prior to smoothing, gaps between individual fibers (red arrow) display a significantly lower intensity than the fiber active area. Periodic variation in contrast is more clearly visible in the line plot in (c). (b) After smoothing, the intensity difference between individual fiber areas and gaps is reduced, visible in the line plot in (d). (e) Microendoscopy image of normal epithelium and resulting line plot of image intensity; scale bar is $250\mu \mathrm{m}$. Line plot was smoothed and plotted against distance in pixels. Black arrows point to first set of minimum (trough) and maxima (peaks). All peak-to-trough values in the image are averaged to yield a global QR value.

Fig. 7

Flow diagram of image preprocessing and quantitative image quality control. The original image was masked and contrast enhanced; then line profiles were extracted every 40 pixels, in a grid-like pattern. Scale bar is $250\mu \mathrm{m}$. The local minima and maxima of each line profile was extracted, and the algorithm calculated the ratio between each minima and its adjacent maxima. The global average of the ratios calculated across the image was termed as the image’s QR.

In order to demonstrate the effect of QR on image quality and overall automated crypt segmentation, we performed the automated QR calculation on the entire porcine image data set (100 images) and human data set (39 images). For each image, in both data sets, the number of crypts was determined via manual inspection to determine a ground truth value. The average QR was tabulated against crypt detection sensitivity (see Secs. 2.6.1 and 3.1) to compute the optimum image exclusion threshold value. The aggregate average sensitivity was calculated by averaging the sensitivity of all images with a QR greater than, or equal to, the respective QR value on the $x$-axis, further described in Sec. 3.2.

2.6.

Automated Segmentation of Crypt Features

QFEA (Fig. 8) was developed to automatically identify and segment each crypt present within microendoscopic images, and use this segmented map of crypt location to compute morphologic parameters such as area and circularity (Sec. 2.6.2). The accuracy of the QFEA’s image segmentation and crypt identification was greatly influenced by the contrast enhancement applied prior to conversion into a binary image (further explained in Sec. 2.6.1). To increase sensitivity of image segmentation, fivefold cross-validation was performed using the porcine (training) data set in order to compute the average detection sensitivity for different contrast enhancement parameters (Sec. 2.6.2). The selected contrast enhancement contrast limits were then applied to both porcine and human images for calculation of sensitivity and PPV of images with various QR values (results in Sec. 3.2).

Fig. 8

Quantification algorithm flowchart. $x$ is the quantified area of each crypt in pixels. Circularity is defined in Eq. (3). Multiple contrast limits were applied to increase sensitivity. Scale bar is $250\mu \mathrm{m}$.

2.6.1.

Selection of optimal contrast limits for image segmentation, using porcine image test data set

As previously stated, all images were preprocessed to reduce common intrascene variation, by masking and contrast enhancement. Section 2.5 describes the preprocessing contrast enhancement, which used a lower contrast limit value of 0 and upper contrast limit value of 1, hereafter referred as the [0 1] contrast limits. Due to contrast intensity variations in images, further contrast enhancement is necessary to create a binary image of crypts with acceptable crypt detection sensitivity. As seen in Fig. 9, intensity-adaptive contrast enhancement was used to improve overall crypt detection accuracy. Contrast enhancement, which requires selection of lower and upper contrast limits, scales image intensity values (0 to 255, for our images) between 0 and 1, whereby intensity values in the image are mapped to new values so that 1% of the image pixels are saturated at the lower and upper contrast limits, as described in Sec. 2.5. Figure 9 shows the histograms of an original image and images after contrast enhancement with different contrast limit pairs. Crypt detection is not shown for the original image, and the image with [0 1] contrast limits, because the image segmentation algorithm (QFEA) did not detect any crypts (neither true positive nor false positive) for either of these images. Note that the QFEA detected crypts in the image that was contrast enhanced using the [0.4 0.9] contrast limits, which were missed by the [0.5 0.9] contrast limits. Contrast limits affect the ability of the QFEA to recognize and segment crypts in different regions of the image; therefore, the overall crypt detection sensitivity of the algorithm was increased with the combination of multiple contrast limit pairs.

Fig. 9

Comparison of image histograms and crypt detection using different contrast limit pairs. Contrast enhancement requires a lower and an upper contrast limit; three contrast limit pairs are shown here [0 1], [0.4 0.9], and [0.5 0.9]. The algorithm did not detect any crypts when using the original image, nor the preprocessed image. Note that the image that was contrast enhanced using the [0.4 0.9] contrast limits detects crypts missed by the [0.5 0.9] contrast limits, thereby increasing the sensitivity of the algorithm with the combination of multiple contrast limit pairs. Scale bar is 250 μm.

To determine the optimal combination of contrast limit pairs for image segmentation, we iteratively increased the values of the lower and upper contrast limits by 10% (Fig. 10). There were 36 contrast limit pairs in the range from [0.1 0.2] to [0.8 0.9]. Due to extremely low crypt detection sensitivity in images with [0 1] contrast limits, the lower contrast limit 0 and the upper contrast limit 1 were excluded from testing.

Fig. 10

Schematic of intensity contrast limit combinations tested for crypt segmentation. Contrast enhancement scales image intensity values (0 to 255) between 0 and 1, scale bar is 250 µm. Both upper and lower contrast limits were iteratively increased by 10% to cover a range from [0.1 0.2] to [0.8 0.9]. Gray region depicts unusable contrast limit pairs.

A training data set comprising 100 images, acquired from porcine colorectal mucosa, was used to assess the ideal intensity contrast limit pairs to optimize crypt segmentation sensitivity. All 36 upper / lower contrast limit combinations were applied to each image, and the sensitivity and PPV of each contrast limit pair was calculated using the number of true positive, false positive, and false negative crypts detected, as compared to manual inspection of the image. Fivefold cross-validation was used to define test and training sets; one fifth of the data was set aside as the test set, and the remaining four fifths were used to train the algorithm. The algorithm selected the best contrast limits using the training set, and then those contrast limits were applied to the test set to calculate crypt detection sensitivity. This was repeated four additional times, so that each of the five groups served as a test set, and resulting output displayed the selected contrast limits and the average sensitivity of the five test sets. The fivefold cross-validation algorithm was repeated 10 times, with the random assignment of images yielding slightly varying sensitivity averages. Sensitivities of several combinations of contrast limit pairs were also calculated using the fivefold cross-validation algorithm; selection of these combinations is described in Sec. 3.1. The average sensitivity, and standard deviation, of each selected contrast limit pair and combination was tabulated.

2.6.2.

Iterative contrast limits and automated crypt identification, segmentation, and morphology quantification

Using the optimal contrast limit pair combinations (determined using the previously described process), porcine and human preprocessed (masked and contrast-enhanced as previously described) individual images and human image mosaics were quantified with QFEA. The quality control step was not used for the initial analysis, of all porcine and human images, in order to determine the optimal QR exclusion value. The QFEA contained the list of selected contrast limit pairs, and passed an image and first set of contrast limits to a quantification subroutine that returned the coordinates of centroids located, and calculated crypt area and circularity. These steps were repeated iteratively (see loop in Fig. 8) with five different pairs of contrast limits, to increase the sensitivity of QFEA. The last step was to save the quantified data and display the centroids (last step in Fig. 8).

Using the optimal contrast limit pair combinations selected after fivefold cross-validation of the crypt segmentation algorithm, as described in Sec. 2.6.1, each of the porcine (100) and human (39) preprocessed images were converted into a binary image, using a variable threshold determined by Otsu’s method,³¹ choosing a threshold that minimizes the variance of black (value 0) and white (value 1) pixels. The binary image was then eroded, with a disk structuring element of 5 pixels radius. Erosion, in short, changes binary values to 0 unless the structuring element, in this case the disk, overlaps only with binary values of 1 when translated to different positions.³² Since we are quantifying circularity of relatively circular crypts, our structuring element is necessarily round and of a radius of 5 pixels since our individual fibers are $\sim 5\text{pixels}$ in diameter.

The resulting binary image was then inverted, with all 1 values converted to 0, and all 0 values converted to 1, and remasked using the field-of-view (FOV) binary image, to ensure that the crypts are the value-1 binary shapes. Holes were then closed, where if a region of value 0 is completely enclosed by value-1 pixel, the region is converted into value-1s as well. The resulting shapes were once more eroded with a disk (5 pixels radius), and the coordinates of centroids of detected shapes were extracted, hereafter referred to as crypts, as well as each crypt’s area and circularity. The centroid for each crypt is the coordinates for the center of mass of each shape. In the eroded binary image, each crypt is defined as a group of continuous value-1 pixel, and the centroid is determined as the average of those pixel positions based on the equations

Eq. (1)

$$C_x=\sum _{i=1}^n{X}_i,$$

Eq. (2)

$$C_y=\sum _{i=1}^n{Y}_i,$$

where $C$ is the centroid coordinate, $(x,y)$ is the position of a value-1 pixel, and $n$ is the number of value-1 pixels in the crypt³³. Crypt area was quantified as the actual number of pixels in the shape (binary value of 1, in this case). The circularity was calculated using

Eq. (3)

$$\text{Circularity}=\frac{4\pi A}{P^2},$$

where $A$ is the area in pixels and $P$ is the perimeter, and a perfect circle has a circularity of 1. After extracting crypt area, crypt circularity, each crypt was quality controlled to exclude those larger than 100,000 pixels in area or smaller than 500 pixels in area (areas outside the range of crypt sizes based on trained observer quantification), or if the circularity was $<0.5$. This exclusion eliminated many shapes that were incorrectly labeled as crypts by the iterative contrast enhancement and erosion steps. Finally, each coordinate was compared against the array of the previous contrast limit pairs’ centroid coordinates (if applicable), and if any two coordinates were at a distance closer than 50 pixels, they were considered duplicates and the duplicate centroid data were not stored. The resulting arrays of crypt area, circularity, and centroid coordinates, as well as the FOV, were returned to the main function.

2.6.3.

Image stack preparation and mosaicking

As described in Sec. 2.4, consecutive images were acquired by slowly dragging the distal end over the epithelial surface for subsequent processing into mosaics. To develop a mosaic image, images from each location (image stack) were cleared of duplicates and images with obvious motion blur. The remaining images in the stack were then preprocessed by masking and contrast enhancement, as described in Sec. 2.5. The image stacks were then stitched together; our mosaics consisted of four to six consecutive images stitched together for a final FOV ranging 3 to 5 mm in length and $\sim 0.7\mathrm{mm}$ in height. Mosaics were created from individual images using a modified version of an existing algorithm,³¹ which requires manual input to select pairs of matching points on two adjacent images, and then uses bilinear interpolation to assign appropriate intensity values in spaces created by stretching or shrinking the second image to match the selected coordinates. Once image stacks were converted into a mosaic, they were treated the same way as a preprocessed individual image and subject to the image segmentation algorithm (QFEA).

3.

Results

Individual mages for this manuscript were acquired from histologically normal regions of human and porcine colorectal tissue, as described in Secs. 2.2 and 2.3. One hundred individual images were acquired from porcine colorectal mucosa as an algorithm training data set and were used in the selection of the best five contrast limit pairs for image segmentation. Thirty-nine images of human epithelial tissue used for automated sensitivity quantification were acquired from clinically normal regions surrounding grossly visible tumors, in excised tissue of eight patients, five diagnosed with invasive colonic adenocarcinoma, two with invasive rectal adenocarcinoma, and one with tubular adenoma. Images of human epithelial tissue used for image mosaicking included both normal and abnormal regions in the same excised tissue of eight patients. All porcine (100) and human (39) images were used for calculating the relationship between average image sensitivity and the QR ratio metric, and determining the optimal QR exclusion threshold value. The images with passing QR were used for quantification of human crypt features in a pilot study, and each individual crypt, as identified by QFEA, was considered a separate data point for quantitative comparison. This comparison is not comprehensive, considering our small sample size, but it shows the ability of QFEA to distinguish variations in crypt size and circularity. A few mosaics were also qualitatively described, as well as subject to the image segmentation and quantification algorithm as proof-of-concept.

3.1.

Selection of Optimal Contrast Limits for Image Segmentation, Using Porcine Image Test Data Set

For improved crypt detection sensitivity during image segmentation, optimal contrast limit pair combinations were selected using fivefold cross-validation, as described in Sec. 2.6.1. The average sensitivity at each of the 36 contrast limit pairs (Fig. 10) is shown in Fig. 11(a) as a heatmap, with color corresponding to the average percent sensitivity at each contrast limit pair. As described previously, multiple contrast limit pairs increase the sensitivity of QFEA. Figure 11(b) shows the average of the contrast limits with the highest test sensitivity (labeled in Fig. 11 as “1”), as calculated via fivefold cross validation, as well as the average and standard deviation of various combinations of contrast limit pairs. Figure 11(a) shows “1” as the contrast limit pair with the highest stand-alone sensitivity and “2” as the contrast limit pair with the second highest sensitivity; combining contrast limit pairs with the same lower contrast limit (contrast limits labeled “1” and “2” both have a lower contrast limit of 0.5) does not yield as large of an increase in overall sensitivity of the QFEA as combining the best contrast limit pair from each row/lower contrast limit (contrast limits labeled “1” and “3”). As seen in Fig. 11(b), combining the two contrast limit pairs with the highest sensitivity increases the overall test sensitivity by $<15\%$, but combining two contrast limit pairs with different lower contrast limits [from different rows in Fig. 11(a)] yields an increase in overall sensitivity of over 35%. A combination of five contrast limit pairs, the highest sensitivity from each row (lower contrast limit) from 0.2 to 0.7, yielded the highest overall test sensitivity [Fig. 11(b)] before a plateau in performance. While adding more than five contrast limit pairs might slightly increase the average reported sensitivity of QFEA, there is the risk of overtraining the algorithm to be too specific to the training set.

Fig. 11

(a) Heatmap of the average percent sensitivity for each contrast limit pair; colorbar is percent sensitivity. As with Fig. 10, the $x$-axis displays the upper contrast limit, and the $y$-axis displays the lower contrast limit. (b) Average test sensitivity, after 10 iterations of fivefold cross-validation, for each contrast limit pair or combination of contrast limit pairs. Maximum standard deviation among all points was $<0.61\%$. Heatmap (a) has numbered the contrast limit pairs selected for average sensitivity calculation, for clear correlation to the $x$-axis in plot (b).

3.2.

Quantitative Quality Ratio and Crypt Detection Sensitivity of the Image Segmentation Algorithm

All porcine (100) and human (39) microendoscopy images were quantified (QFEA) using the five contrast limit pairs selected in Sec. 3.1. The quantitative image quality control algorithm, described previously, yielded a global QR metric (peak-to-trough ratio) per image, and this average ratio was tabulated along with the crypt detection sensitivity of QFEA, for each image. In Fig. 12(a), the sensitivities and PPVs of images with a QR value equal to, or higher than, the specified QR value ($x$-axis) were averaged for an aggregate average sensitivity at that QR. The peak aggregate sensitivity, before the decline, was at a QR value of 1.07 [Fig. 12(a)]. Therefore, a QR value of 1.07 was selected as the image quality control threshold, excluding for quantification any images with a QR of $<1.07$. QR values $>1.1$ used $<10$ images for average sensitivity calculation, and are not reliable indicators.

Fig. 12

(a) Average sensitivity and PPV of the QFEA crypt detection versus the average QR of images; aggregate performance plotted against QR was calculated using images with QR values equal to or higher to the QR labeled in the $x$-axis. (b) Number of images with QR values equal to or higher than the labeled QR values.

3.3.

Pilot Study of Quantified Crypt Morphology

Quantified crypt data were compared between images of surrounding grossly appearing normal regions in patients grouped as case 1 or case 2; case 1 were patients diagnosed with invasive adenocarcinoma and case 2 were patients diagnosed with tubular adenoma. For individual images, crypt area and circularity were significantly greater for the case 2 group than the case 1 group. The tabulated area and circularity, displayed in Fig. 13 in arbitrary units, were divided by the maximum data value in area and circularity, respectively. A total number of 10 images with 219 cumulative identified crypts were used for case 1, and nine images with 219 cumulative identified crypts for case 2. A two-tailed Student $t$-test yielded $p$ values of $<0.01$ for the area, and $<0.005$ for the circularity.

Fig. 13

Comparison of image features in individual microendoscopic images of clinically normal tissue in patients grouped as case 1 and case 2. Case 1 had an adjacent region diagnosed as invasive adenocarcinoma and case 2 had an adjacent region diagnosed as tubular adenoma. Error bars denote standard error, and $p$ values were calculated using two-tailed Student $t$ test. Microendoscopy images on the right are examples of quantified images. Scale bar is 0.5 mm.

3.4.

Qualitative Image Analysis and Automated Crypt Identification of Mosaics

Mosaics were created from consecutive fields of view to demonstrate the ability to improve the inherently small field of view in fiber bundle microendoscopy images, and the image segmentation and quantification algorithm was applied to them to determine if quantification of the mosaics using our algorithm was viable. Figure 14 shows a qualitative comparison of image mosaics acquired from both normal and dysplastic regions of colonic epithelium. The original images were acquired over 10 s, and the new FOV was $\sim 3.5\mathrm{mm}$ long and 0.7 mm wide for both mosaics. The mosaics comprise five individual consecutive images. Normal tissue [Fig. 14(a)] has regularly spaced crypts, surrounded by a pericryptal sheath, demonstrated here as a bright ring enclosing the luminal compartment (see yellow arrow). Neoplastic tissue [Fig. 14(b)] oftentimes demonstrates irregularly shaped crypts that are haphazardly arranged, and the yellow arrows point to crypts that have uneven ring structures enclosing them. The crypt detection sensitivity, when applied to the mosaics in Fig. 14, was 70% [Fig. 14(b)] or higher [Fig. 14(a)].

Fig. 14

Image mosaics showing (a) normal colon and (b) invasive adenocarcinoma. Yellow arrows point to crypts. Images have enhanced contrast and brightness for publishing. Scale bar is 0.5 mm.

Figure 15 shows an occult hyperplastic polyp in otherwise normal-appearing tissue, $\sim 1\mathrm{mm}$ in diameter, whose relative size and morphology was highlighted by mosaicking. A stack of eight images was mosaicked using the method stated previously, with the yellow arrows [Fig. 15(a)] pointing to the edges of the hyperplastic polyp $\sim 1\mathrm{mm}$ in diameter. The imaged region was $\sim 3\mathrm{mm}$ in length and 0.7 mm in width—the yellow arrows on gross tissue [Fig. 15(b)] point to the edges of the imaged region. The white arrow [Fig. 15(b)] points to the grossly visible tumor, which was $\sim 4\mathrm{cm}$ to the right of the hyperplastic polyp [yellow arrows, Fig. 15(b)].

Fig. 15

Example of occult dysplasia. (a) Image mosaic showing a hyperplastic polyp. Yellow arrows point to edges of hyperplastic polyp. Scale bar is 0.5 mm. (b) Context image showing gross view of colon epithelium and location of image mosaic at left. Yellow arrows point to mosaic endpoints ($\sim 3\mathrm{mm}$). White arrow points to cancerous region (invasive colonic adenocarcinoma). Scale bar is 10 mm. Images have enhanced contrast and brightness for publishing.

4.

Discussion

Fiber bundle microendoscopy images of ex vivo colorectal epithelium contain useful quantitative data that may be used to assess the tissue microstructure for evidence of occult dysplasia. In this paper, we have described QFEA, an approach to automated quantification of important features of colorectal epithelial microstructure from microendoscopy image data. One hundred images of normal porcine tissue were used to select, via fivefold cross-validation, the best contrast limit pair combinations for crypt detection sensitivity. In addition, we have also demonstrated a method to automatically calculate, and exclude, images with low signal-to-background ratios (QR) in order to filter and remove images that may yield unreliable quantitative data. In clinical settings, the fiber bundle microendoscopy imaging system used in this study may acquire image data via manual placement of the probe directly into the epithelial surface, or remote placement onto tissue during an endoscopic procedure. In both of these applications, significant image quality variability may occur. The majority of the images demonstrated a crypt detection sensitivity of $>80\%$ following analysis via QFEA, indicating potentially reliable calculation of quantitative reporters of tissue microstructure.

A combination of contrast limit pairs was tested with the porcine image data set to determine the best parameters for high crypt detection sensitivity. While the contrast limits were iteratively increased by 10%, it is likely that smaller step increases would have provided increased overall sensitivity. A larger data set would be required to test if smaller iterative increases and contrast limit selection would generalize for microendoscopy images, or if the algorithm would be overtrained. The crypt identification algorithm, using the selected combination of contrast limit pairs, was then applied to both porcine and human images, to determine the aggregate performance when excluding images with low QR (Fig. 12). As seen in Fig. 12(b), the number of images, both porcine and human, with QR values $>1.1$ drops below 20, decreasing sample size and leading to undersampling bias in the calculated sensitivities at those values. Figure 12(a) reveals that the PPV of human images is drastically lower than that of the porcine image set. We believe this is due to a more carefully controlled imaging procedure with the porcine tissue, since image quality of human tissue is influenced by variations in human preoperative procedures, and sample handling by clinicians and staff. Potential alternatives to improve image quality include alternative contrast agent or optical sectioning techniques such as structured illumination.³⁴

A pilot comparison of quantified crypt morphology (Sec. 3.3) demonstrated the ability of our quantitative image quality control and image feature extraction algorithm (QFEA) to detect minor differences between two otherwise clinically normal appearing colorectal tissue specimens. While a conclusive statement cannot be made based on image data from eight human patients, it does suggest that an algorithm such as QFEA can quantify crypt features objectively and without significant user input. Similarly, no definite quantitative statements can be made for image mosaics, until a larger data set is acquired. Qualitative inspection of the image mosaics shows them as a viable method for overcoming the FOV size limitations of microendoscopy, while image segmentation of the mosaics shows the potential applicability of QFEA to mosaicked images.

In vivo inspection of gastrointestinal mucosa using high-resolution microendoscopy methods has been previously shown to yield clinically useful data. Recent studies examining manual inspection of microendoscopy images of known diagnoses have demonstrated strong interobserver agreement, indicating that well-trained clinicians should be able to effectively incorporate these methods into clinical practice.²⁰ Qualitative assessment of these images by a highly trained observer is likely to continue to yield clinically relevant data, but widespread adoption of high-resolution, in vivo, microendoscopy methods for improved detection of occult dysplasia remains a challenge. A user-independent method of image analysis could provide additional data to make an informed decision and reduce the burden of training required for the clinician at the point-of-care. The introduction of an image analysis algorithm like QFEA into computer-aided diagnostic methods may yield useful complementary data to make improved clinical decisions. In our pilot study, QFEA demonstrates an ability to detect minor differences between two otherwise clinically normal appearing colorectal tissue specimens, suggesting that discriminative quantitative crypt features can be measured objectively and without significant user input.

Computer-aided diagnostic approaches have been of significant research interest in myriad cancer detection approaches, including imaging. Automated quantitative classification has been applied to breast cancer detection using ultrasound images, digitized mammograms, and histopathology, ovarian cancer detection using matrix-assisted laser desorption/ionization imaging mass spectrometry, and prostate cancer grading using ultrasound images and histopathology.^35–40 Quantitative classification based on microendoscopy image texture features in Barrett’s esophagus and esophageal squamous cell carcinoma, specifically gray-level co-occurrence matrices, has been previously investigated.^41–43 A recent publication on automated selection of high-quality frames in microendoscopic images⁴⁴ points to both the need for quality control, as well as the viability of automated extraction of high-quality frames during video-rate acquisition. Ongoing development work with microendoscopic systems may enable spectroscopic data to be used in conjunction with quantitative image features, such as those described in this paper.⁴⁵ This quantitative data, coupled with extracted image-based features provided by algorithms such as QFEA, could lead to the adoption of standardized, objective methods of in vivo high-resolution microendoscopy image interpretation and analysis.

In conclusion, the methods described here provide insight into the ability of fiber bundle based high-resolution microendoscopy imaging tools to yield clinically relevant data about the tissue microstructure in colorectal epithelium. Effective clinical translation and application of any quantitative image analysis tools applied to this imaging method will need to account for some unavoidable image quality variation, particularly for in vivo endoscopic application of the device on living, constantly moving tissue. Quantitative assessment and control of high-resolution image quality is essential to ensuring reliable results in any future objective, computer-aided diagnostic algorithm development.