1.

Introduction

Chest x-ray images could be used in an automatic image analysis for detecting and classifying abnormalities, including diseases of atelectasis, consolidation, infiltration, pneumothorax, edema, emphysema, fibrosis, effusion, pneumonia, pleural thickening, cardiomegaly, nodule mass, and hernia, as mentioned in the NIH chest x-ray dataset.¹ On top of these diseases, which are commonly diagnosed using chest x-ray images, this paper focuses on the new disease of COVID-19 which causes lung damage and can also be detected in chest x-ray images.² Several methods were proposed recently to address this research domain. Existing methods are reviewed below. All of them were developed based on the convolutional neural network (CNN).

Zhang et al.³ adopted 18 layers of a residual CNN that was pretrained with ImageNet dataset. The sigmoid was used as an activation function, with the binary cross-entropy loss function. Differently, Wang et al.⁴ trained the model to classify a chest x-ray image into three classes including normal, COVID-19, and other viral and bacterial diseases. Its model was constructed using five convolutional layers of first-stage projection, expansion, depthwise representation, second-stage projection, and extension. Narin et al.⁵ attempted using three different types of CNN architectures, including ResNet50, InceptionV3, and Inception-ResNetV2. It was found that the pretrained ResNet50 using ImageNet provided the highest classification performance. While, Apostolopoulos and Mpesiana⁶ attempted using five different types of CNN architectures, including VGG19, MobileNet v2, Inception, Xception, and Inception ResNet v2. However, their research reported the best performance on using MobileNet v2 with the transfer learning of ImageNet.

Hemdan et al.⁷ tried seven different types of CNN architectures, including VGG19, DenseNet121, InceptionV3, ResNetV2, Inception-ResNet-V2, Xception, and MobileNetV2. They recommended two models, VGG19 and DenseNet201, which achieved the best classification accuracy. Afshar et al.⁸ proposed a CNN architecture containing four convolutional layers and three capsule layers, by taking three-dimensional x-ray images as the input. The model could classify four labels of normal, bacterial, non-COVID viral, and COVID-19. Abbas et al.⁹ used a CNN model of DeTraC transformation pretrained using ImageNet as the backbone architecture. Principal component analysis was used to reduce the dimension of the extracted feature, and the nearest centroid based on the squared Euclidean distance was applied as the main classification. Li et al.¹⁰ attempted using three lightweight CNN architectures of MobileNetV2, ShuffleNetV2, and SqueezeNet. Karim et al.¹¹ reported the best performance by combining VGG-19 and DenseNet-161.

Similarly, Minaee et al.¹² trained their model by trying four CNN architectures, including ResNet18, ResNet50, SqueezeNet, and DenseNet-121. The SqueezeNet provided the best performance. Mangal et al.¹³ used the CheXNet based on 121-layer DenseNet which was trained on ChestX-ray14.¹⁴ In addition, Khan et al.¹⁵ also developed a COVID-19 classification model using CNN with the inception pretrained based on ImageNet dataset. Similarly, Hall et al.¹⁶ and Bukhari et al.¹⁷ both adopted Resnet50 pretrained using ImageNet dataset.

As can be seen from existing methods in the literature, they were developed using CNN-based approaches with various well-known architectures. Their results are reported and compared in Sec. 3. They used pretrained networks that were trained on either the popular ImageNet dataset or the related chest x-ray images dataset, such as ChestX-ray14. Therefore, they had to limit the size of input images to be the same size as that used in the pretrained models, which was usually small. Another important reason for using a small size of input image is a limited computational resource of RAM and GPU. This may not be able to cope with a small-sized signal of COVID-19 in a chest x-ray image.

This paper proposes a solution using ResNet-101, 101 layers deep.¹⁸ Although it has a version with pretrained weights on ImageNet database,¹⁹ the proposed method adopts ResNet-101 as a backbone architecture but is trained from scratch. This is mainly because the pretrained model was learned on a small size input image of $224\times 224\text{pixels}$, which may fail to detect some small signals of the COVID-19 in a chest x-ray image. This paper develops a solution model by taking a large-sized input image of $1500\times 1500\text{pixels}$.

Importantly, in our experiments, the proposed method is evaluated using four subdatasets of D1: COVID-19, D2: normal (without any diseases and remarks), D3: other normal (without any diseases but with some remarks), and D4: other abnormal (with other diseases). Our models are trained for two versions of (1) classifying into two classes of COVID-19 and non-COVID-19 and (2) classifying into three classes of COVID-19, normal, and other normal together with other abnormal.

In this paper, the experimental results are reported in terms of confusion matrix, accuracy, sensitivity, and specificity, under different cut-off thresholds of the confidence value in the output layer of the model. Heatmaps emphasizing signals of COVID-19 in chest x-ray images are computed from filtering kernels of the model. They are visualized in the region of segmented lungs using U-Net, since COVID-19 is supposed to be within the lung region in chest x-ray images.

The rest of this paper is organized as follows. Section 2 explains the proposed method of classifying COVID-19 in chest x-ray images. Section 3 illustrates the experimental results in various scenarios. Then, results are discussed in Sec. 4, and conclusions are summarized in Sec. 5.

2.

Materials and Methods

Figure 1 shows an overview framework of the proposed solution. The training, validating, and testing chest x-ray images are resized into $1500\times 1500\text{pixels}$.²⁰ Then, the real-time data augmentation is applied on the original training images. Both original and augmented training images are fed to train the classification model based on the backbone architecture as explained below. The trained model is then validated with the original validating images. If the validating result is converged or the maximum number of epochs is reached, then the training and validating processes are stopped and the final model is concluded. Otherwise, it goes back to the data augmentation process and repeats to the next epoch.

Fig. 1

Overview framework of the proposed method.

In the testing phase, the trained model is applied on each resized chest x-ray image, to compute predicted class, its confidence score, and heatmap. The details are explained in the following sections.

2.1.

Backbone Architecture

The ResNet-101^21,22 is adopted in our proposed method as the backbone architecture of the COVID-19 classification model. It is a very deep network containing deep layers as shown in Table 1,²³ with more than 44 millions parameters.

Table 1

The convolutional layers in ResNet-101 used in the proposed solution.

Layer name	Building block of (kernel size, filter)	Number of building blocks
conv1	$[7\times 7,64]$	1
conv2	$\left[\begin{array}{c}1\times 1,64\\ 3\times 3,64\\ 1\times 1,256\end{array}\right]$	3
conv3	$\left[\begin{array}{c}1\times 1,128\\ 3\times 3,128\\ 1\times 1,512\end{array}\right]$	4
conv4	$\left[\begin{array}{c}1\times 1,256\\ 3\times 3,256\\ 1\times 1,1024\end{array}\right]$	23
conv5	$\left[\begin{array}{c}1\times 1,512\\ 3\times 3,512\\ 1\times 1,2048\end{array}\right]$	3

As shown in Table 1, each square bracket represents each building block of convolutional layers which are parametrized by kernel size and filter. For example, the kernel size of $7\times 7$ means the height and width of the two-dimensional convolution window are both 7. While, the filter refers to the dimensionality of the output space or the number of output filters in each convolutional layer.²⁴ The third column represents a number of defined building blocks used in each convolutional layer. For example, in the layer named “conv5,” each building block contains three convolutional layers connected in sequence as shown in Fig. 2.

Fig. 2

A building block in conv5 of ResNet-101.

The input to the network is a chest x-ray image, as shown in Fig. 1. The ResNet-101 has a version with pretrained weights using ImageNet dataset which is a large-scale classification dataset containing 1.2 million training images from 1000 classes of objects.²² However, the input image’s size must be limited to the pretrained requirement of $224\times 224\text{pixels}$. This may not cope well in the case of differentiating normal class from other-normal classes.

In this paper, the ResNet-101 is trained from scratch using the input images of the large size $1500\times 1500\text{pixels}$. The top part (i.e., classification part) of ResNet-101 is replaced with the global average pooling, softmax, and output layers. Five types of data augmentations are added on the training dataset, including zoom, rotate, shear, flip, and shift.²⁵

The proposed solution develops two types of models which are different in the output layer. The first model is developed to classify COVID-19 class from any non-COVID-19 class, having two nodes in the output layer. While, the second model is developed to classify a chest x-ray image into three classes having three nodes in the output layer of COVID-19, normal without any diseases, and norther normal with some other diseases or remarks.

3.

Results

This section explains and discusses our experimental results on different scenarios. Both our own dataset and published dataset⁴ are used in the experiments, as shown in Table 3. For the published dataset, only chest x-ray images with COVID-19 are used in our experiments, because they are used to validate the cross-datasets scenario of COVID-19 detection.

Table 3

Datasets used in the experiments to validate the proposed solution.

In addition, in our D1 dataset, the 142 images of COVID-19 cases were obtained from three levels of the severity as: (1) 22 images of the severe level, (2) 13 images of the moderate level, and (3) 107 images of the mild level. Each patient case has only one image taken in each instance. In the training and testing processes, each individual image is fed as the input into the CNN-based model at a time.

Five datasets, as listed in Table 3, are used in our four scenarios of experiments as below. The results are reported in terms of confusion matrix, accuracy, sensitivity, and specificity.

Images from the five datasets (D1 to D5) are independently split into two subsets of (1) training and validating set and (2) testing set, as mentioned in each scenario. Later, the training and validating set is further randomly split into the training set and validating set with proportions of 90% and 10%, respectively, in each epoch of the CNN training phase. So, in all cases, images in training, validating, and testing sets are independent and nonoverlapped. The numbers of independent images in training, validation, and testing arrangements of each scenario are summarized in Table 4.

Table 4

The numbers of independent images from the five datasets (D1 to D5) used in training, validation, and testing arrangements of each scenario.

In this paper, the positive class is drawn when the confidence value predicted by the trained CNN-based model is higher than the cut-off score. The descriptive statistical analysis is used to determine the results in terms of sensitively, specificity, and accuracy. These performances are also compared with other existing methods in the literature.

3.6.

Heatmaps and Confidence Scores

As shown in Fig. 1, the heatmap is computed for each test chest x-ray image to emphasize high-weight signals of COVID-19. The filters’ weights on the final convolutional layer are extracted to compute the final heatmap. Sample filters’ weights of one test chest x-ray image are shown in Fig. 3. In this example, the high-weights (i.e., yellow color) are located around lungs’ regions, because COVID-19 could damage lungs.

Fig. 3

Sample filters’ weights of a test chest x-ray image.

Then, the final heatmap is generated by averaging these filters’ weights. It is computed for individual test chest x-ray image. Sample final heatmaps are shown in Fig. 4. The first three heatmaps are computed from chest x-ray images with COVID-19. It can be seen that the high-weights of yellow patches are regions detected by the trained model to be signals of COVID-19. The medical experts can concentrate on these regions to final check the disease, while the final heatmap is computed from a chest x-ray image with the normal label. There are thus no yellow patches to represent any COVID-19 damage.

Fig. 4

Sample final heatmaps of individual test chest x-ray images.

Our solution also generates a confidence score of being COVID-19 for each test chest x-ray image. Four examples of chest x-ray images with COVID-19 and four examples of chest x-ray images with non-COVID-19 are shown in Fig. 5. It is clearly seen that the test images with COVID-19 could be classified correctly to be COVID-19 with very high confidence scores of above 90%. Also, the test images with non-COVID-19 could be classified correctly with very low confidence scores to be COVID-19 of below 30% or, in other words, with very high confidence scores to be non-COVID-19 of above 70%.

Fig. 5

Sample final heatmaps with their confidence scores of being COVID-19. (a) Four examples of chest x-ray images with COVID-19 and (b) four examples of chest x-ray images with non-COVID-19.

4.

Discussion

The original trained model using the proposed method can classify a chest x-ray image into two classes of COVID-19 and non-COVID-19. The training and validating samples of the non-COVID-19 class are normal chest x-ray images without any remarks or diseases. This model is shown to achieve high performance on testing COVID-19 and normal chest x-ray images. This is mainly because the patterns of COVID-19 and normal cases are seen in the training process. However, its performance is significantly dropped when it is tested with chest x-ray images with other remarks or diseases such as fibrosis, spondylosis of spine, tuberculosis, pneumonia, and pulmonary edema.

This is as expected because patterns of other remarks or diseases are not seen and learned in the training process. Also, they occur in lungs’ regions as similar to COVID-19. So, they could be easily confused with COVID-19, using this developed model. It can result in many false detection/positive cases, which could not be acceptable for practical usages.

Therefore, the model is further improved by adding sample chest x-ray images containing other remarks and diseases in the training and validating processes. This makes the model to learn differences between patterns of COVID-19 and patterns of other diseases. It results in increasing the specificity score and reducing the false detection of COVID-19. However, the sensitivity score is also lower, when compared with the original trained model.

Then, the next version of the developed model is trained to classify a chest x-ray image into three classes of COVID-19, normal, and other diseases. This could maintain the sensitivity score and increase the specificity score. This is because separating the other diseases class from the non-COVID-19 class can reduce the confusion between COVID-19 and other diseases, and the confusion between normal and other diseases. As shown in Fig. 5, the COVID-19 cases are clearly separated from the non-COVID-19 cases (i.e., normal and other diseases), with very high confidence scores.

As additionally reviewed by the expert, the generated heatmaps of the COVID-19 cases could identify areas of COVID-19 correctly. However, some heatmaps of the false-positive cases are reported incorrectly as shown in Fig. 6. The first two images are normal cases and the last image contains another abnormality. The confidence scores of being COVID-19 of the three images are all higher than 50%. However, to be classified as COVID-19 with the cut-off of 90%, only the second image is wrongly classified. In addition for these three cases, the heatmaps incorrectly highlight the non-COVID-19 cases as COVID-19 (i.e., yellow areas).

Fig. 6

Sample final heatmaps with their confidence scores of the false-positive cases. The first two images are normal cases and the last image contains another abnormality.

However, none of the developed models can achieve a perfect performance of 100% accuracy. Thus, they should be adopted for the prefiltering of normal cases, by cutting off chest x-ray images that are classified to be COVID-19 with very low scores—that is, they have high confidence to be non-COVID-19. In this way, it can be used to reduce a number of chest x-ray images that must be manually diagnosed by human experts.

In addition, the heatmap is generated to emphasize possible areas of being COVID-19 in each chest x-ray image. This can be an assistive tool for human experts to be used together with the computed confidence score, to conclude the final diagnosis.

5.

Conclusions

This paper presents a solution for COVID-19 classification in chest x-ray images. Its backbone CNN architecture is developed using ResNet-101. The model is trained from scratch with a large size of the network’s input of $1500\times 1500\text{pixels}$. Data augmentation is also applied on the original training images to enhance the regularization of the model. It is developed in two versions of classification: two-classes-based and three-classes-based. The two-classes-based version is used to classify chest x-ray images into COVID-19 and non-COVID-19. The three-classes-based version is used to classify chest x-ray images into COVID-19, normal, and other abnormal. The proposed solution achieves very promising sensitivity, specificity, and accuracy of 97%, 98%, and 98%, respectively. The developed solution can also generate the heatmap with a confidence score of being COVID-19, to emphasize the result on each test image. The heatmap is visualized on only lung regions segmented using U-Net.