2.1.

Image Enhancement

In this paper, we use the BCISC¹⁵ public dataset, and Fig. 2 shows some of the leukocyte images in the BCISC dataset. We find that the background color and brightness of the blood cell images are not uniform, resulting in color and brightness variability among the leukocytes as well, presumably due to the influence of lighting and the surrounding environment during the acquisition process. To eliminate this effect, we propose an AHE-Retinex correction for image pre-processing, combining AHE and multiscale Retinex with color restoration (MSRCR),¹⁶ which can better balance the color of leukocytes in different images and further clarify their features. In addition, one image in the BCISC dataset contains only one leukocyte, which does not meet the requirements of this paper. Therefore, in this paper, after AHE-Retinex correction of all images, 4 random images of these leukocytes were formed into $512\times 512\text{pixel}$ size images, as shown in Fig. 3, with a total of 50 images. This will initially form the data set required for the study in this paper.

Fig. 2

Selected leukocyte images from the BCISC dataset: (a) basophils, (b) eosinophils, (c) lymphocytes, (d) monocytes, and (e) neutrophils.

Fig. 3

The final processed dataset: (a) original random four images and (b) final result images.

The basic idea of HE is to widen the gray levels with a large number of pixels in the image and reduce the gray levels with a small number of pixels, to achieve a clear image.¹⁷ The general HE input is only grayscale images, but the blood cell images in this paper are RGB color images, so this paper first performs the RGB to HSV operation on the blood cell images and HE on the V channel. Considering the fineness of the leukocyte features, the direct use of the traditional HE will lead to the loss of most of the feature information. Therefore, to solve this problem, AHE is used in this paper.

The algorithm underlying retinex theory is single scale retinex (SSR), which is implemented in the steps of Algorithm 1.

Algorithm 1

Single scale retinex.

Multiscale Retinex (MSR) is to select three scale parameters in step 2 of SSR to form three Gaussian surround functions and then convolve them separately and weigh the average to get the illumination estimation components of each channel to effectively maintain the detail and color information. MSRCR, on the other hand, is based on MSR and uses the color recovery function in step 4 to multiply with the MSR enhancement function of each channel. MSRCR is based on MSR, and the color recovery function is multiplied by the MSR enhancement function of each channel in step 4 to obtain the image enhancement reflection component of the three channels to reduce the color bias. The AHE-Retinex algorithm proposed in this paper is based on MSRCR, converting the image in RGB color space to HSV color space, and performing AHE processing on the V component. The overall algorithm processing process is shown in Fig. 4.

Fig. 4

AHE-Retinex correction process.

The MSRCR model is as follows:

2.2.

Skip-Connection

The key to the U-Net⁵ network structure is the skip-connection between the contracting path and the expanding path, which combines the deep high-level features from the expanding path with the shallow low-level features from the contracting path. The contracting path progressively down-samples the feature map through the pooling layer, while the expanding path up-samples the low-resolution feature map into a pixel-level segmentation result map. To compensate for the information lost in downsampling during the encoding stage, the U-net network uses a hopping connection to fuse the feature maps at the corresponding positions in the two processes, so that the expanding path can obtain more high-resolution information when up-sampling, and thus better recover the information in the original image. Therefore, adding the attention mechanism to the skip connection enables the network to pay more attention to the important parts of the shallow features in the upsampling process at each level, and through the fusion of the shallow features with the deep features, the network can retain more high-resolution detail information contained in the shallow feature maps, thus improving the image segmentation accuracy. Then considering that the spatial information and the edge information in the image are very important in the leukocyte segmentation task, this paper adopts a CBAM¹⁰ approach that connects CAM and SAM in tandem, and its implementation schematic is shown in Fig. 5.

Fig. 5

CBAM.

It can be seen that CBAM contains two separate submodules, the CAM and the SAM, which perform channel and spatial attention, respectively. This not only reduces parameters and the amount of calculations, but also ensures that it can be integrated into existing network architectures as a plug-and-play module.

The CAM is shown in Fig. 6. The input feature map ($H\times W\times C$) are subjected to global max pooling and global average pooling based on width and height, respectively, to obtain two $1\times 1\times C$ feature maps. Then, they are fed into a two-layer neural network (MLP). The number of neurons in the first layer is $C/r$ ($r$ is the reduction rate), the activation function is Relu, and the number of neurons in the second layer is C. This two-layer neural network is shared. Then, the MLP output features are subjected to element-wise summation operation and then sigmoid activation operation to generate the final channel attention feature. Finally, the obtained channel attention feature and the input feature map are multiplied element-wise to generate the input features required by the SAM.

Fig. 6

CAM.

That is, CAM compresses the feature map in the spatial dimension to obtain a one-dimensional vector and then operates on it. When compressing in the spatial dimension, not only the average pooling but also the max pooling is taken into account. The average pooling and max pooling can be used to aggregate the spatial information of the feature map, send it to a shared network, compress the spatial dimensions of the input feature map, and sum up element by element to produce a channel attention map. The average pooling has feedback for every pixel point on the feature map, while the max pooling has feedback for gradients only where the response is greatest in the feature map when performing gradient backpropagation calculations.

The SAM is shown in Fig. 7. The feature map output from the CAM is used as the input feature map for this module. First, global max pooling and global average pooling based on channels are performed to obtain two $H\times W\times 1$ feature maps. Then, these two feature maps are concatted based on channels (channel stacking). Then, the spatial attention feature is generated by sigmoid. Finally, this feature is multiplied by the input feature to obtain the final generated feature.

Fig. 7

SAM.

Again, the SAM is to compress the channels, and the average pooling and the max pooling are performed in the channel dimension. Max pooling operates by extracting the maximum value on the channel, and the number of extractions is height × width; the average pooling operates by extracting the mean value on the channel, and the number of extractions is also height × width.

2.3.

Loss Function

The use of loss functions usually takes into account the characteristics of the dataset. Usually there is a problem of extreme class imbalance in the blood cell images, which means that the number of different types of leukocytes can vary greatly, for example, the number of eosinophils and basophils is very small. The small number of samples produces fewer loss values, forming a hard-to-score sample; while the majority of the total loss function input parameters are easy-to-score samples, thus making the optimization direction of the model (the gradient descent direction of the loss function) not as desired in this paper. For this problem, the focal loss function can be used, with the following equation:

In addition, in the experiments of leukocyte segmentation, we found that the nuclei of different types of leukocytes can be easily segmented because of their different morphologies. However, the cytoplasm was more difficult to be segmented. First, the cytoplasm of leukocytes is similar in color to red blood cells in the stained blood cell images, so it is easy to segment the red blood cells together. Second, the cytoplasm of different types of leukocytes is also similar, which may cause confusion. Therefore, in this paper, considering the relevant a priori knowledge that the cytoplasm of neutrophils, eosinophils, and basophils contains special staining granules after staining with Richter’s dye, while monocytes and lymphocytes have no cytoplasmic granules, Dice loss is used to further regulate the loss function by the different granules on the cytoplasm of leukocytes. The equation of Dice loss is as follows:

In general, the direct use of Dice loss will adversely affect the back propagation and easily make the training unstable, while this paper then combines focal loss to solve the class imbalance problem of white blood cells in blood cell images. The loss function in this paper is as follows:

3.1.

Datasets

In this paper, we used four datasets named Dataset1, Dataset2, Dataset3, and Dataset4. Dataset1¹⁵ is from the Third People’s Hospital of Fujian Province, China, and contains 268 individual leukocyte images of $256\times 256\text{pixels}$ in size (51 neutrophils, 54 eosinophils, 56 basophils, 54 monocytes, and 53 lymphocytes), which were processed into 50 multiple leukocyte images of $512\times 512\text{pixels}$ in size to suit the experimental purpose of this paper. Dataset2²¹ was collected by Jiangxi Dekang Technology Co., Ltd., in China and contained 300 individual leukocyte images of $120\times 120\text{pixels}$ (176 neutrophils, 22 eosinophils, 1 basophils, 48 monocytes, and 53 lymphocytes). Dataset3²¹ was published in the CellaVision blog and contains 100 individual leukocyte images of $300\times 300\text{pixels}$ in size (30 neutrophils, 12 eosinophils, 3 basophils, 18 monocytes, and 37 lymphocytes). Dataset4,²² called LISC, is a peripheral blood image of healthy subjects and contains 242 individual leukocyte images of $720\times 576\text{pixels}$ in size (50 neutrophils, 39 eosinophils, 53 basophils, 48 monocytes, and 52 lymphocytes). The images in Dataset2 were captured under fast staining conditions, while the images in the other three datasets were captured under standard staining conditions. In this paper, these datasets were re-labeled with different colors to indicate different types of leukocytes, as shown in Fig. 8.

Fig. 8

Labeling colors of leukocytes.

3.2.

Evaluation Metrics

To evaluate the segmentation performance and robustness of the image, this paper introduces the concept of a confusion matrix.²³ Among them, TP is true positive, which indicates that positive samples are correctly determined as positive samples; FP is false positive, which indicates that negative samples are misjudged as positive samples; FN is false negative, which indicates that positive samples are misjudged as negative samples; and TN is true negative, which indicates that negative sample is correctly determined as a negative sample. In this paper, precision ($P$), recall ($R$), F-measure (F1), intersection over union (IoU), and accuracy (Acc) are selected. Precision and recall are mutually constrained, and it is difficult to judge the performance of segmentation from these two metrics alone, while the comprehensive metric F1 considers precision and recall, which can evaluate the performance of the algorithm more comprehensively. Among them, if the IoU obtained from each class of prediction is summed and averaged, mean intersection over union (mIoU) will be obtained. Then the higher the values of F1, IoU, Acc, and mIoU, the higher the segmentation accuracy. These indicators are defined as follows:

3.3.

Experimental Environment

The experiments in this paper are based on the deep learning framework TensorFlow, and the experimental environment is Python 3.6. The processor is i7-9700f, the memory is 8G, the graphics card is GTX1660 Ti, and the operating system is Windows 10. The dataset used in this paper is the BCISC public dataset, provided by the Third People’s Hospital of Fujian Province, China. There is only one leukocyte on a single image in this dataset. To achieve the purpose of multiple leukocyte segmentation, four images were randomly taken for the stitching process in this paper. There are 50 multiple leukocyte images of size $512\times 512\text{pixels}$ available. First, the sample dataset was divided into training and test sets in the ratio of 9:1. Then, the training set is fed into the U-net network for model training. In the experiment, all the training data were normalized to $512\times 512\times 3$, and the pretrained weights were the initialized weights of the VGG16 network, and then the training started from generation 0. Because the training weights were used to train the VGG16 weights, to put more resources on training the network parameters of the later part, the VGG16 backbone part was first frozen and trained for 50 generations, and then all of them were thawed and continued to be trained together for 50 generations, which makes the time and resource utilization can be greatly improved. Two data samples are crawled at a time. The initial learning rate is set to $1\times {10}^{-4}$.

4.1.

Accuracy Analysis

To verify the accuracy and effectiveness of the proposed method in this paper, comparison experiments were conducted on Dataset1.¹⁵ Three classical segmentation methods, U-Net,⁵ DeepLabv3+,¹⁸ and PSPnet,¹⁹ were used for the comparison. The comparison of evaluation metrics is shown in Table 1. From Table 1, it can be seen that the method in this paper performs better compared to the other three segmentation methods. Among the three classical segmentation methods, we found that for each category of IoU, eosinophils, lymphocytes, and monocytes differed more compared to the other two categories of cells. This is due to the fact that each of these three types of leukocytes has certain differences of its own. For example, eosinophils it has a nucleus similar to neutrophils and is easily fragmented, and their eosinophilic granules can be dispersed around the cells, which makes network segmentation somewhat difficult. In human blood, lymphocytes are mainly composed of small lymphocytes and a certain number of medium lymphocytes, then these two types of lymphocytes are morphologically different, and the network will have some difficulty in identification. Then monocytes are due to their nucleus being kidney-shaped or horseshoe-shaped and other polymorphic, the cells also have the problem of different shapes, for the network to learn there is a certain degree of difficulty. Then in this paper, data augmentation, skip connection and loss function improvement are performed for these problems. The background of the blood cell image is formed a strong contrast with the leukocyte itself by data enhancement, so that the broken eosinophils dispersed out of the eosinophils can also be well captured by the network, and the characteristics of each leukocyte can also be enhanced. Adding an attention mechanism to the skip connection can better enable the network to focus on the different features of each type of leukocyte, thus improving the discrimination ability of the network to some extent. Then changing the loss function can improve the feature weights of the hard-to-score samples, thus solving the problems caused by the hard-to-score samples. Then from the final mIoU obtained by the method of this paper, it seems that this paper basically solves the problem of appeal, and the maximum difference of mIoU for each kind of leukocytes is at 3.13%. From the value of F1, it seems that the model output of the method in this paper is the best result. The best results were obtained for neutrophils, mainly because neutrophils have the most remarkable characteristics, with rod-shaped nuclei or $2\sim 5$ foliated nuclei and thin filaments connected between lobes, which are different from the characteristics of all other types of leukocytes.

Table 1

Comparison of metrics of different segmentation methods on Dataset1.

Figure 9 shows the comparison of the style results of three classical segmentation methods, U-Net,⁵ DeepLabv3+,¹⁸ and PSPnet,¹⁹ and the method in this paper. Among them, Fig. 9(a) shows the original image stitched from four random images in the original Dataset1; Fig. 9(b) shows the image after AHE-Retinex processing; Fig. 9(c) shows the masked image, which serves as a comparison with the final segmentation result; and Fig. 9(d) shows the segmentation result of U-Net network. According to the results, U-Net network has better segmentation effect for leukocytes, but there is evidence of misclassification problems. Since the cytoplasm of lymphocytes and monocytes does not have granules and other features, the U-net network is confusing for the cytoplasm of lymphocytes and monocytes. Figure 9(e) shows the segmentation result of DeepLabv3+ network, which is poor for leukocyte segmentation and has the problems of missing and mis-detecting; most of the leukocytes cannot be identified and segmented. The two leukocytes on the right can be segmented in the first image because of the obvious and unique characteristics of basophils (containing basophilic granules) and neutrophils (with rod-shaped nuclei or 2 to 5 lobes and thin filaments between lobes). Figure 9(f) is the segmentation result of PSPnet network; the edges of the leukocytes segmented by this network are smooth, and the original shapes of most leukocytes are lost. Also, there are serious problems of leakage and false detection. Figure 9(g) is the segmentation result of the method in this paper. The result graph can show more intuitively that the segmentation effect of the method proposed in this paper is better, and not only can it accurately segment different kinds of leukocytes but also can segment the edge shape of leukocytes.

Fig. 9

Comparison of segmentation results of different segmentation methods on Dataset1 dataset: (a) original image, (b) AHE-Retinex processed image, (c) masked image, (d) U-Net network segmentation results, (e) DeepLabv3+ network segmentation results, (f) PSPnet network segmentation results, and (g) segmentation results of this method.

4.2.

Robustness Analysis

To verify the robustness of the proposed method in this paper, four datasets are used in this paper to conduct comparison experiments on different methods. The datasets we used are Dataset1¹⁵, Dataset2,²¹ Dataset3,²¹ and Dataset4,²² which are all four datasets with only one leukocyte present on one image; the compared methods are FCN,²⁰ U-Net,⁵ WBC-Net,⁸ and LeukocyteMask¹⁵ and the proposed method in this paper, respectively. The evaluation metrics are P, R, F1, mIoU, and Acc, and the data comparison is shown in Table 2. From Table 2, it can be seen that both in F1 and mIoU and in Acc, the approach in this paper achieves good performance compared to the other four approaches. Although it performs poorly on Dataset4 compared to the other three datasets, the other four approaches do not achieve better accuracy either. We studied these four datasets and found that the range of leukocytes on the blood cell images in Dataset4 was small and the leukocyte features were blurred. Considering the four datasets together, the model output of this paper’s method is better and the segmentation accuracy achieves good results. Overall, the performance of this paper’s method for leukocyte segmentation is the best.

Table 2

Comparison of the metrics of different segmentation methods on different datasets.

4.3.

Ablation Experiment

A total of three partial improvements are proposed in this paper, which is summarized in Table 3, with mIoU and Acc selected as comparative metrics. Among them, the original network is a U-net with a loss function of the CE loss function, and both metrics are improved to a great extent after adding the AHE-Retinex image enhancement. With the addition of CBAM or replacing the loss function with a combination of Dice loss and focal loss, another improvement was achieved, indicating that the features of leukocytes in blood cell images are more complex and difficult to segment. Finally, all three improvements were used in U-net for leukocyte segmentation, and a good result was obtained. mIoU could reach 91.89% and Acc could reach 99.53%.

Table 3

Ablation experiment.

The final leukocyte segmentation results are shown in Fig. 10(a) shows the original image of any three stitched images in the dataset; Fig. 10(b) shows the three images after AHE-Retinex processing; Fig. 10(c) shows the label mask corresponding to these three images, which serves as a comparison with the final segmentation results; and Fig. 10(d) shows the segmentation results of the U-net with loss function CE. According to the results, the original network has the problem of misclassification and inaccurate judgment for the segmentation of leukocytes. Since there are no features such as granules on the cytoplasm of both lymphocytes and monocytes, the U-net with loss function CE is confusing for the cytoplasmic judgments of lymphocytes and monocytes. The lymphocytes in the fourth image were misjudged as monocytes; Fig. 10(e) shows the segmentation result of the data after AHE-Retinex processing. Since the characteristics of each type of leukocyte are clearer, some improvement was achieved in the problem of cytoplasmic confusion, but a small number of misjudgments occurred, mainly in the second image where the nuclei of neutrophils were misjudged as eosinophils, because the nuclei of eosinophils and neutrophils are too similar, and the network is more likely to confuse the two when the characteristics of leukocytes are clear. The misjudgment situation of the fourth image still exists; Fig. 10(f) is the result of the segmentation of CBAM added to AHE-Retinex. The results show that the nucleus confusion problem in Fig. 10(e) still exists but has been somewhat improved, greatly reducing the range of misclassified pixels. The percentage of monocytes and lymphocytes being misclassified is also decreasing, but the problem still exists in the third and fourth images; Fig. 10(g) is the segmentation result of replacing the loss function with a combination of Dice and focal functions based on AHE-Retinex. It is found through the results that the change of the loss function greatly improves the prediction results of the hard-to-score samples. The medium and fine granular cells in the second figure can be detected and segmented completely. Then for the hard-to-score lymphocytes and monocytes also achieved some improvement, the misclassified range in the third graph basically no longer exists, and the misclassified range in the fourth graph has been significantly reduced; Fig. 10(h) is the segmentation result of the proposed method in this paper, which combines all three methods together. The problems in Figs. 10(d) and 10(e) are solved, the cytoplasmic misclassification problem is solved, and the most difficult lymphocytes in the fourth graph are also segmented, resulting in a better leukocyte segmentation result.

Fig. 10

Leukocyte segmentation results from the ablation experiment. The purpose of the ablation experiment is to verify the validity of the proposed improvement points, and the fusion of the predicted result of the masked and original images gives a more effective indication of the effectiveness of the improvement points. The leukocyte image segmented by removing the background does not illustrate the problem, e.g., even if the edge parts of monocytes and lymphocytes are confused, as long as there are masked parts they can be segmented intact. So in comparison to Fig. 9, part labels (d), (e), (f), (g), and (h) in Fig. 10, there are no additional segmented leucocyte images: (a) original image, (b) AHE-Retinex processed image, (c) masked image, (d) segmentation result of U-net network with loss function of CE, (e) segmentation results of the data after AHE-Retinex processing, (f) segmentation results of AHE-Retinex with CBAM added on top of it, (g) the segmentation result of replacing the loss function with a combination of Dice and focal functions on the basis of AHE-Retinex, and (h) segmentation results of the method in this paper.

4.4.

Additional Experiment

To verify the generality and effectiveness of the method proposed in this paper, dataset1 was randomly combined again in this paper, so that 4, 9, and 16 leukocytes were present on the $512\times 512$ size image. The final segmentation results are shown in Fig. 11.

Fig. 11

Segmentation results of different numbers of leukocytes. (a) The original image and segmentation results with 4 leukocytes, (b) original figure and segmentation results with 9 leukocytes, and (c) the original image and segmentation results with 16 leukocytes.

From the experimental results, we found that the segmentation accuracy is higher on images containing fewer leukocytes, while a small amount of cytoplasmic misclassification occurs as the number of leukocytes increases, but in general it does not segment the whole leukocyte. As shown in Fig. 11(a), when using the $2\times 2$ size stitching, the image contains four leukocytes, and the experimental results show that its segmentation accuracy is high and it is suitable for segmentation containing multiple types of leukocytes. When the $3\times 3$ size is used, the number of leukocytes increases to 9, as shown in Fig. 11(b), and good segmentation results can be achieved for both similar and different types of leukocytes. However, when the number of leukocytes in the image increased to 16 using the $4\times 4$ size stitching, there was an error in the performance of multiclass leukocyte segmentation. As shown in the second image in Fig. 11(c), the cytoplasm of the second leukocyte on the leftmost side of this image shows mis-segmentation and it is not difficult to find that although there is a small part of mis-segmentation, it is not significant for the overall segmented image of leukocytes. Also, the first image in Fig. 11(c) shows that the model proposed in this paper has a high accuracy rate for multiple numbers of the same type of leukocytes. Therefore, through the above experimental analysis, the method proposed in this paper has some generality and validity.

In addition, in this paper, the network of contracting paths was changed to ResNet50²⁴ to further validate the generalizability of the improved U-net, and all the improvement points mentioned in the previous section (image enhancement, CBAM, and loss function) were left unchanged. Table 4 shows a comparison of the mIoU and Acc data for the VGG16 and ResNet50 networks. resNet50 also achieved good results, with a segmentation accuracy of 95.18%. However, directly replacing the contracting path with the ResNet50 network did not achieve better results than VGG16.

Table 4

Network comparison experiments for contracting path.