2.1.

Celiac Disease

CD is a multisystemic immune-mediated disease, which is associated with considerable morbidity and mortality.¹⁴ In untreated or inappropriately treated CD, the inflammation caused by the dysregulated immune response can disrupt the intestinal mucosa thus leading to a total atrophy of the villi (finger-like projections of the mucosa) which causes a diminished ability to absorb nutrients. More than two million people in the United States, this is about one in 133, have this disease.¹⁵

For the automated diagnosis of CD, we differentiate between healthy mucosa and mucosa affected by CD using an image database consisting of images gathered by the modified immersion technique (MIT¹⁶) using traditional white-light illumination (denoted as ${\mathrm{WL}}_{\mathrm{MIT}}$) as well as narrow band imaging (denoted as ${\mathrm{NBI}}_{\mathrm{MIT}}$). Examples of the two classes for both imaging modalities are shown in Figs. 1(a)–1(d). It was shown that using ${\mathrm{NBI}}_{\mathrm{MIT}}$ or ${\mathrm{WL}}_{\mathrm{MIT}}$ as imaging modality has a significant impact on the underlying feature distribution of general purpose image representations.¹⁷ However, it was also shown that systems trained on images from both modalities generalize well without requiring additional domain adaption techniques and that combining both modalities improves the accuracies in case of an insufficient amount of data for training (as is probably the case for CNNs).

Fig. 1

In the top row, we show example images of the CD database for the two classes healthy and CD using ${\mathrm{NBI}}_{\mathrm{MIT}}$ as well as ${\mathrm{WL}}_{\mathrm{MIT}}$ endoscopy. In the bottom row, we show example images of both classes of CP from the HD database (left side) and the HM database (right side). (a) ${\mathrm{NBI}}_{\mathrm{MIT}}$, CD, (b) ${\mathrm{NBI}}_{\mathrm{MIT}}$, healthy, (c) ${\mathrm{WL}}_{\mathrm{MIT}}$, CD, (d) ${\mathrm{WL}}_{\mathrm{MIT}}$, healthy, (e) HD neoplastic, (f) HD non-neoplastic, (g) HM neoplastic, and (h) HM non-neopl.

Prior works dealing with the computer-aided diagnosis of CD employed general purpose image descriptors such as local binary pattern (LBP)-based operators^18,19 as well as wavelet-transform based features.^20,21 Furthermore, methods specifically designed for the diagnosis of CD were developed such as shape features²² describing the curvature in mucosal structure, bandpass-type Fourier filters²³ extracting features from bandpass filtered images, and fractal analyze-based features.¹⁰ In general, contrast sensitive image descriptors turned out to be well suited for the automated diagnosis of CD because the most distinctive visual difference between healthy and abnormal images is the partly or entirely missing of the villi in case of areas affected by CD. Thus, images of affected areas have less contrast than images showing healthy mucosa [see Figs. 1(a)–1(d)]. Recently, CNNs were applied for the diagnosis of CD and outperformed previously used image descriptors.^2,24 There is also an survey on computer-aided decision support for the diagnosis of CD.²⁵

2.2.

Colonic Polyps

CPs are a rather frequent finding and are known to either develop into cancer or to be precursors of colon cancer. Hence, an early assessment of the malignant potential of such polyps is important as this can lower the mortality rate drastically.

For the automated diagnosis of CP, we differentiate between two classes, normal mucosa or hyperplastic polyps (class non-neoplastic) and neoplastic, adenomatous, or carcinomatous structures (class neoplastic). Experiments are applied on two CP databases. One CP image database is gathered by high-definition (HD) endoscopy in combination with the i-Scan technology (further denoted as HD database) and the second CP image database is gathered by high-magnification (HM) endoscopy in combination with chromoscopy (further denoted as HM database). Example images of the two classes for both CP databases are shown in Figs. 1(e)–1(h).

In the following, we give an overview on prior works on the computer-assisted staging of CP using HD endoscopic image databases or HM endoscopic images databases:

3.1.

CNN Training and Feature Extraction

This section gives the implementation details for CNN training and CNN feature extraction as well as the description of the employed nets.

In this work, three different CNNs are employed, the AlexNet,³⁴ the VGG-f net,¹ and the VGG-16 network.³⁵ AlexNet and the VGG-f net both consist of five convolutional layers and three FC layers with a final SoftMax classifier. The VGG-16 net consists of 13 convolutional layers subdivided in five convolutional blocks (where each of the two to three convolutional layers inside of a block has the same number and sizes of filters) and three FC layers with a final SoftMax classifier.

As already mentioned in Sec. 1, we apply two CNN learning strategies.

For the first learning strategy, the CNNs are trained on a specific endoscopic image database. As initialization for the convolutional layers of the AlexNet and VGG-f net, we use the parameters that were learned on the ImageNet data. In case of the VGG-16 net, the convolutional parameters are randomly initialized³⁶ as the VGG-16 net did not work well using pretrained coefficients as initialization. As the FC layers are more specific to the details of the classes contained in the ILSVRC ImageNet challenge data, we randomly initialize the coefficients of those layers instead of using the parameters that were learned on the ImageNet data. The size of the last FC layer is adapted to our two-class classification scheme, which means that the size of the convolutional filters is changed from $1\times 1\times 4096\times 1000$ for the ImageNet to $1\times 1\times 4096\times 2$. The last FC layer is acting as SoftMax classifier and computes the training loss (log-loss). Training is performed on batches of 128 images each, which are for each iteration randomly chosen from the training data and subsequently augmented (see Sec. 3.4). Stochastic gradient descent with weight decay ($\lambda =0.0005$) and momentum ($\mu =0.9$) is used for the training of the models.

For the second learning strategy, we use the CNNs that were pretrained on the ImageNet ILSVRC challenge data as fixed feature extractors without any training on our CD or CP datasets.

For both learning strategies, the activations of CNN convolutional layers are extracted as features for further Fisher encoding. That means the images are fed through the networks and the outputs of convolutional layers [without pooling or nonlinearization (ReLU)] are used for Fisher encoding.

3.2.

Fisher Encoding of CNN Activations

This section gives the implementation details for applying IFV encoding on the activations of CNN convolutional layers.

IFVs¹² are based on estimated Gaussian mixtures of local image descriptors. The improved version based on a nonlinear Hellinger’s kernel and $l^2$ normalization with 16 clusters is used. Each activation of a convolutional layers consists of $K$ different $M\times N$ ($M=N$ in our case) feature maps, where $K$ is the number of filters kernels in the convolutional layer and $M\times N$ is the size of the filter response for each filter kernel. These feature maps can be viewed as $N\times M$ local descriptors of dimension $K$. The IFV image representation is obtained by pooling those $M\times N$ local descriptors for a given activation of a convolutional layer. The IFV representation of an convolutional layer is a vector of length $K\times 2\times 16$ [16 Gaussian mixture model (GMM) clusters and two parameters (mean and covariance)]. Figure 2 shows the pipeline of our CNN-Fisher approach from the CNN feature acquisition to the Fisher representation and to the final SVM classification.

Fig. 2

Scheme of our proposed CNN-Fisher approach.

The reason why we use 16 GMM clusters for Fisher encoding is that the number of local CNN features is already quite low for the last convolutional layer [169 ($M=N=13$, $M*N=169$) for VGG-f and AlexNet and 196 ($M=N=14$, $M\times N=196$) for VGG-16] and hence a too high number of clusters would not make sense. We want to use the same number of clusters for each convolutional layer and so using 16 clusters is an appropriate choice. We also perform experiments with 8 and 32 clusters, but, if not mentioned otherwise, 16 clusters are used.

We do not only classify the extracted IFV features of single convolutional layers, we also combine the information of several different convolutional layers by concatenating the IFV feature vectors of those layers. More specifically, in case of the AlexNet and VGG-f net we separately classify the IFV representation of each layer yielding in one classification result per layer. Additionally, we concatenate the IFV representations from the first $i$ convolutional layers with $i\in \{2,\dots ,5\}$ and further classify the concatenated feature vectors [and so we get one result for the combination of the first two convolutional layers ($i=2$), one for the first three ($i=3$), one for the first four ($i=4$), and one for the combination of all convolutional layers ($i=5$)].

In case of the VGG-16 net, we proceed analogously and separately classify the Fisher representations of the first convolutional layer of each convolutional block (the VGG-16 net consists of five convolutional blocks with two to three convolutional layers per blocks) and additionally we concatenate the Fisher vector representations of the first convolutional layers from the first till the $i$’th convolutional block with $i\in \{2,\dots ,5\}$ for further classification.

So, for example, the IFV representation of the second convolutional layer of the VGG-16 net is 4096 dimensional [$128\times 2\times 16$ ($K=128$, 2 parameters and 16 GMM clusters)] and the concatenation of the IFV representations from all five layers is 47,104 dimensional [$(64+128+256+512+512)\times 2\times 16$].

3.3.

Comparison Methods

To compare the results of our proposed CNN-Fisher approach with results of other approaches, we additionally perform experiments using five CNN-based approaches and four handcrafted image descriptors. The nine methods are either state of the art in the automated diagnosis of CD or CP or they are related to the CNN-Fisher approach. For the five CNN-based approaches, we apply the same nets as for our proposed approach to enable a direct comparison with the CNN-Fisher approach. All comparison methods except of one (CNN SoftMax Classification) are classified using linear SVMs.

3.4.

Experimental Setup

The CD database consists of 1661 RGB image patches of size $128\times 128$ pixels that are gathered by means of flexible endoscopes using ${\mathrm{NBI}}_{\mathrm{MIT}}$ as well as ${\mathrm{WL}}_{\mathrm{MIT}}$. About 1045 images are gathered by ${\mathrm{WL}}_{\mathrm{MIT}}$ endoscopy (587 healthy images and 458 affected by CD) and 616 images are gathered by ${\mathrm{NBI}}_{\mathrm{MIT}}$ endoscopy (399 healthy images and 217 affected by CD). So, in total 986 image patches show healthy mucosa and the remaining 675 image patches show mucosa affected by CD. The images were captured from 353 patients. As our used nets require input image sizes of $224\times 224\times 3$ (VGG-f net and VGG-16 net) respectively $227\times 227\times 3$ (AlexNet), the images of the CD database are bicubicly upscaled to a size of $256\times 256\times 3$ to be able to extract patches of the images for training and classification.

The HD database was acquired by extracting patches of size $256\times 256\times 3$ from frames of HD-endoscopic (Pentax HiLINE HD+ 90i Colonoscope) videos. Our database consists of patches gathered with four different imaging modalities [three different i-Scan modes (modes 1, 2, and 3) and no i-Scan]. The HD database consists of 478 image patches (144 images showing non-neoplastic mucosa and 301 images showing neoplastic polyps) from 84 patients.

The HM database is acquired using a zoom-colonoscope (Olympus Evis Exera CF-Q160ZI/L) with a magnification factor of 150 and indigocarmine dye-spraying. The database is acquired by extracting 716 patches (198 non-neoplastic and 518 neoplastic) of size $256\times 256\times 3$ from 327 endoscopic color images of 40 patients.

The images of all three databases are extracted only from regions for which histological findings are available.

The image data is normalized by subtracting the mean image of the training portion for each database. We then linearly scale each image within $[-\mathrm{1,1}]$.

Due to the small amount of available data, we use data augmentation to increase the number of images for the training of the nets. For each training iteration, a batch of images is randomly chosen from the training data. The augmentation is based on cropping a subimage [$227\times 227$ (AlexNet), respectively, $224\times 224$ pixels (VGG-f and VGG-16)] from each image of a batch with randomly chosen position. Subsequently, the subimage is randomly rotated (0 deg, 90 deg, 180 deg, or 270 deg) and randomly either flipped or not flipped around the horizontal axis. Training is performed for 5000 (AlexNet and VGG-f net) and 10,000 (VGG-16 net) iterations, respectively. Validation is performed using the central patch of an image and no data augmentation techniques are applied to the validation images.

Because of the relatively small amount of data, we perform fivefold cross validation to achieve a stable estimation of the generalization error. For each of the fivefolds, we took care that images of a single patient are never in training and evaluation sets. All nets and comparison methods are trained using the training portion of our data corpus and we use the same folds for each method. The final validation is performed on the left-out part. That means for each network architecture and each image database, we train five different nets, one for each of the fivefolds (except for the case where we use the CNNs as fixed feature extractor, where no training is performed). In our experiments, we compute the overall classification rate (OCR) for each fold and report the mean OCR over all fivefolds. For the comparison of the results from our CNN-Fisher approach with the results of other state-of-the-art approaches, we additionally report the standard deviation and the range of OCRs over the fivefolds.

For each of our databases, we use the same training and evaluation set portions for all methods.

The CNNs are implemented using the MatConvNet framework³⁷ and the SVMs are provided by the LIBLINEAR library.³⁸ We use a linear one-class SVM and the SVM cost factor (C) is found using cross validation on the training data. SVM classification is applied to our proposed method as well as to all comparison methods except for the CNN SoftMax classification. We use the implementation of IFV as provided by VLFeat.

4.1.

Results of the Convolutional Neural Network-Based Methods

The results of the experiments using our three nets trained on the endoscopic image databases, respectively, on the ImageNet database (IN) are shown in Table 1. Our proposed CNN-Fisher approach is applied using either the Fisher representations of single (CNN-Fisher SL) and multiple convolutional layers (CNN-Fisher ML), whereat $C_i$ denotes the $i$’th convolutional layer and $C_{1,\dots ,i}$ denotes the combination of the first $i$ convolutional layers. As comparison methods, we employ the five CNN-based methods SoftMax classification (SM), the SVM classification of FC layer activations (FC), and the BoW, VLAD, and sum approach. We only show the results of the comparison approaches with the same training approach [training on the endoscopic image database or using a pretrained net (IN)] as used in the literature. The results for the other (not computed) training approaches are marked with a “−”. For each CNN, the best result over the different classification approaches is given in bold face numbers. To provide an overview of which CNN-based methods work best in general for endoscopic image classification, we additionally present the averaged results over the results of all databases and nets per method ($Ø$ in Table 1). Once again we differentiate between nets trained on endoscopic image databases (E-IMDB) and nets trained on the ImageNet database (IN).

Table 1

Mean classification rates over the fivefolds. The CNNs are trained on a specific endoscopic database (the CD, HD, or HM database) or on the ImageNet database (IN). In the two bottom rows (Ø), we show the method’s average results over all databases and net architectures [three (databases) × three (net architectures) = nine results per method] for training on the endoscopic image databases (E-IMDB) as well as for training on the ImageNet database (IN).

As we can see in Table 1, the clearly best results for each net architecture and each database are achieved using our proposed CNN-Fisher approach using multiple convolutional layers and no adaption of the nets to the endoscopic databases. The overall best result for the CD database (92.5%) is achieved using the VGG-16 net trained on the ImageNet database using all convolutional layers ($C_{1,\dots ,5}$), for the HD database the best result (91.2%) is achieved using the VGG-16 net trained on the ImageNet database with layers $C_{1,\dots ,3}$, and for the HM database the best result (93.7%) is achieved using the VGG-f net trained on the ImageNet database with layers $C_{\mathrm{1,2}}$.

So, combining the information of multiple convolutional layers distinctly improves the results compared with just using single layers. The later a CNN convolutional layer is in the net, the bigger is the receptive field size (the region in the input space that a particular CNN’s feature is looking at) of its extracted features. So, the combination of several layers leads to a multiscale representation of an image that clearly outperforms the single-scale representation using only one convolutional layer.

The best results for endoscopic image classification in general are achieved by our CNN-Fisher approach combining the information from three to five convolutional layers ($C_{\mathrm{1,2},3}$, $C_{1,\dots ,4}$, and $C_{1,\dots ,5}$). As we can see in Table 1 [$Ø$ (IN)], these three configurations of the CNN-Fisher approach achieve average classification rates across all databases and nets of over 90% for nets trained on the ImageNet database. That means the number of incorrectly classified images is nearly halved in comparison with the best performing CNN comparison method (sum: 83.2%).

When we consider the CNN-Fisher results using single convolutional layers (CNN-Fisher SL), we can observe that the best results in general are achieved for the first three convolutional layers ($C_1$, $C_2$, and $C_3$). These three configurations of the CNN-Fisher approach achieve average classification rates across all databases and net architectures of about 87% for nets trained on the ImageNet database. So for the purpose of endoscopic image classification, the information contained in the earlier layer activations seems to be more suited than the information in the later convolutional layer activations. Applying Fisher encoding to the last convolutional layer activation $C_5$ (as proposed in Refs. 6 and 7) only achieves an average classification rate of 84.5% and the four CNN comparison methods only achieve average classification rates over all nets and databases of up to 83.2%.

When we take a look at the results of the FC activations, we can observe that training the nets on an endoscopic database generally improves the results compared with using the nets trained on the ImageNet database (like already showed in a previous work about the classification of CD²). Only the VGG-16 net performs better without training on an endoscopic database. We think that the VGG-16 net is too deep (too much parameters to learn) to be properly trained on our relatively small endoscopic images databases. The results of the SoftMax classifier are for all databases similar to those of the SVM classification of FC activations.

The results of the comparison methods BoW, VLAD, and sum are clearly worse than the results of our proposed approach. So, these three approaches combining shallow image representations with CNN activations are not as well suited for endoscopic image classification as our proposed approach. The VLAD approach is pooling the FC layer activations from patches of an image, but the endoscopic images mostly show homogeneous texture structures and so the patches of an image look quite similar. So this approach does make sense for object recognition (for which it was originally proposed), where objects stand out from the background, but apparently not for endoscopic images with their homogeneous texture structure. The BoW approach is quite similar to our proposed approach using the last convolutional layer $C_5$. The BoW approach applies BoW pooling and CNN-Fisher SL ($C_5$) applies Fisher pooling to the activations of the last convolutional layer. So, the only big difference between these two approaches is the pooling method (BoW versus Fisher). The results of CNN-Fisher SL ($C_5$) are about 3% higher in average than those of the BoW approach, which means that Fisher encoding is more suited than BoW encoding for our purpose. The sum approach is applied to the third convolutional layer ($C_3$). By comparing the results of the CNN-Sum approach with those of CNN-Fisher SL ($C_3$), we can see that that the results of CNN-Fisher SL ($C_3$) are $>4\%$ higher in average than those of the sum approach. So, Fisher encoding is more suited for the classification of endoscopic images than the more simple sum approach.

The highest differences in the results of the CNN-Fisher approach between the pretrained and the adapted version of a net architecture occur in case of the VGG-16 net, which was the only net architecture that was fully randomly initialized (contrary to the other two nets, where only the FC layers were randomly initialized and the parameters pretrained on ImageNet where used as initialization for the convolutional layers).

4.2.

Comparison with State-of-the-Art Approaches

In Fig. 3, we compare the results of our proposed CNN-Fisher method using the best performing configurations [multiple layers ($C_{1,\dots ,5}$) and the VGG-16 net trained on the ImageNet database] compared to five state-of-the-art approaches in the automated diagnosis of CD or CP (four shallow image descriptors and the SVM classification of CNN FC layer activations using the VGG-16 net). Our CNN-Fisher approach is additionally applied using 8 and 32 clusters for Fisher encoding (16 is the default parameter) to find out the optimal number of clusters.

Fig. 3

Results of our proposed CNN-Fisher approach with 16 (standard), 8, or 32 clusters [ML(16), ML(8), and ML(32)] compared with the results of five state-of-the-art approaches on our three endoscopic image databases. The bars show the mean classification rates over the fivefolds, the black error bars the standard deviations, and the blue error bars show the range of classification rates over the fivefolds. (a) CD, (b) HD, and (c) HM.

As we can see in Fig. 3, our proposed method clearly outperforms other state-of-the-art methods on all three databases. We can observe that using 16 clusters for Fisher encoding achieves higher results than using only eight clusters. Using a higher number of clusters (32) does only slightly change the results and does not improve them overall.

We can observe in Figs. 3(b) and 3(c) that the classification rates are quite differing across the fivefolds for the two polyp image databases (note the big error bars). This is caused by the low number of patients in these two databases (HD:84 and HM:40). As all images of a patient have to be in the same fold, each fold contains only a very small number of patients. This, together with the fact that the images of some patients are easier/harder to classify than those of other patients, leads to the situation that some folds are easier/harder to classify than others. That is the main reason for the method’s high standard deviations and ranges of values over the fivefolds in case of the two polyp databases.

4.3.

Comparison of the Training Approaches

In Fig. 4, we see the convolutional kernels of the first convolutional layer of the three nets learned on the ImageNet as well as their convolutional kernels after training (the already pretrained nets) on our CD database, which is our image database with clearly the most images (1661).

Fig. 4

Convolutional kernels of the first convolutional layer learned on the ImageNet database or learned on our CD database (CDB). (a) AlexNet ImageNet, (b) AlexNet CDB, (c) VGG-f ImageNet, (d) VGG-16 ImageNet, (e) VGG-16 CDB, and (f) VGG-f CDB.

In case of the VGG-16 net, which was fully randomly initialized, the filter kernels are entirely different for the CD version and the ImageNet version of the net. We can observe in Figs. 4(d) and 4(e) that the CD database is too small to train well-structured and smooth CNN filters and that is probably the reason why our proposed CNN-Fisher approach achieved distinctly higher results using the nonadapted ImageNet version of the VGG-16 net (see Table 1).

In case of the AlexNet and VGG-f net, we can observe that the training on the CD database only slightly changed the filter kernels. Only in case of the AlexNet, some of the filter kernels changed completely. Some of the filters of the AlexNet changed their colors [e.g., the filter on the right side of the third row in Figs. 4(a) and 4(b)] some other lose their entire structure and turned into simple averaging filters [e.g., the left filter in the first row in Figs. 4(a) and 4(b)]. We think that this loosing of structure could possibly be the reason for the higher results of our CNN-Fisher approach (for $C_1$) using the ImageNet version of the AlexNet. In case of the VGG-f net, the filter kernels undergo only very minor changes and also the differences between the CNN-Fisher results (for $C_1$) are only rather small between the adapted and the nonadapted version of the net. All the observations about changes of the filter kernels by training the nets on the CD database (based on Fig. 4) do also apply for the two other endoscopic databases (HD and HM).

So altogether that means our proposed method achieves higher results without training the nets on the endoscopic databases, whereas the widely used approach classifying FC layer activations generally performs better with nets trained on the endoscopic databases. The main reason for this behavior is that our CNN-Fisher approach uses the more generic information from the convolutional layers, whereas the approach classifying FC layer activations uses class-specific information from the FC layers (which requires training to the specific classes of the endoscopic database). In case of the VGG-16 net, an additional reason for the higher results using the nonadapted net is that the net is too deep to be properly trained on our small databases that results in overfitting to the training data and filter kernels that are neither smooth nor well structured.

So, a big advantage of our CNN-Fisher approach compared with the SVM classification of FC activations and the SoftMax classification is that we can save time by omitting the training of the nets.

4.4.

Statistical Significance

By means of the McNemar test,³⁹ we assess the statistical significance of our results from the CNN-based methods. With the McNemar test, we analyze if the images from a database are classified differently by the employed methods or if most of the images are classified identical. The McNemar test examines if the classification results of two methods are significantly different for a given level of significance (we use $\alpha =0.05$) by building test statistics from incorrectly classified images. For the McNemar test, we employ the CNN-based comparison methods and both versions of our proposed approach (single layer and multiple layers) using the best performing combination of layers. That means we employ the second layer for the Fisher representation of single layers [SL ($C_2$) in Fig. 5] and all five layers for the Fisher representation of multiple layers [ML ($C_{1,\dots ,5}$) in Fig. 5]. The McNemar test is applied for the methods using the VGG-16 net trained on the ImageNet database as this net achieved the best results over all three databases. The results of the MCNemar test on our three image databases are shown in Fig. 5. As we can see in Fig. 5, the CNN-based methods are generally significantly different. In case of the CD database, the CNN-Fisher approaches are significantly different than the other CNN-based methods. For the HD database, the CNN-Fisher approaches are significantly different than the other approaches except to each other and the sum approach [SL ($C_2$) compared to sum]. In case of the HM database, the multiple layer Fisher representation is significantly different than the other approaches, whereas the single-layer representation is only significantly different than the VLAD approach and the multiple layer representation.

Fig. 5

Result of the McNemar test for our three endoscopic image databases. A black square in a plot means that the two considered methods are significantly different with significance level of $\alpha =0.05$. A white square means that there is no significant difference between the methods. (a) CD, (b) HD, and (c) HM.