3.1.

Feature Extraction Module by the Combination of Swin Transformer and CNN

Traditional convolutional neural networks excel at extracting local features but are limited by the fixed size of their convolutional kernels, making it challenging to effectively capture global contextual information. This limitation can impact the ability to extract features of clouds and consequently affect the accuracy of cloud detection.

In contrast to CNNs, Swin Transformer employs a hierarchical attention mechanism to efficiently handle global long-range dependencies in images. It decomposes the image into a series of small patches and utilizes multiple layers of transformer modules for information propagation and feature extraction. Within each local window, the features are encoded as vector representations and interact through a self-attention mechanism to capture global dependencies.

To fully utilize the strengths of Swin Transformer and CNNs, we employ the FEM in the encoding stage of the model. The Swin branch captures the global contextual information of images, whereas the CNN branch captures the local texture and detail information. In the FFM, the global and local information is fused.

3.1.1.

CNN branch

The CNN branch is designed to capture local texture details of clouds, and it consists of two parts: encoding and decoding. The encoding part consists of two elements, conv block (CB) and downsample block (DB). CB is composed of a convolution, BatchNorm, rectified linear unit (ReLU) activation function, and basic block. The structure of the basic block is consistent with that of ResNet.³⁰ DB is composed of a downsampling operation and multiple basic blocks. In CB, the convolutional kernel has a size of 3 with a stride of 2 and a padding of 1. In DB1 to DB4, the number of basic blocks is 3, 4, 6, and 3, respectively. The CFEM is incorporated in the middle of the encoding–decoding process to perform multiscale feature extraction. In the decoding part, the features from the encoding part are concatenated and upsampled, gradually restoring them to their original sizes. In UP, the convolutional kernel has a size of 3 with a stride of 1 and a padding of 1. The feature sizes of $f_1$,$f_2$, $f_3$, and $f_4$ are 1/32, 1/16, 1/8, and 1/4 of the original size, respectively. The specific structure of the CNN branch is shown in Fig. 4.

Fig. 4

Architecture of the CNN branch.

3.1.2.

Swin branch

The Swin branch is used to extract the global semantic information of clouds. From Fig. 5, it can be seen that the Swin branch mainly consists of patch partition, patch embedding, patch merging, Swin Transformer block, and CFEM. The Swin branch consists of four layers, each of which extracts features corresponding to different scales to capture semantic information at different levels. In our implementation, the number of Swin Transformer blocks in each layer is set to 2, 2, 6, and 2. The feature sizes of $S_1$, $S_2$, $S_3$, and $S_4$ are 1/4, 1/8, 1/16, and 1/32 of the original size.

Fig. 5

Architecture of the Swin branch.

The Swin branch divides the image into a set of patches using a patch partition. Each patch is then linearly embedded into the Swin Transformer block by patch embedding. Patch merging involves dividing the features equally and placing them in different channels, similar to a pooling operation. The CFEM is utilized for multiscale feature extraction, similar to the CNN branch. The Swin Transformer block consists of four main components: window-based multi-head self-attention, shifted window-based multi-head self-attention, multi-layer perceptron (MLP), and layer normalization. The equations are as follows:

Eq. (1)

$${\stackrel{\mathrm{\Lambda }}{Y}}^l=W-\mathrm{MSA}(\mathrm{LN}(Y^{l-1}))+Y^{l-1},$$

Eq. (2)

$$Y^l=\mathrm{MLP}(\mathrm{LN}({\stackrel{\mathrm{\Lambda }}{Y}}^l))+{\stackrel{\mathrm{\Lambda }}{Y}}^l,$$

Eq. (3)

$${\stackrel{\mathrm{\Lambda }}{Y}}^{l+1}=\mathrm{SW}-\mathrm{MSA}(\mathrm{LN}(Y^l))+Y^l,$$

Eq. (4)

$$Y^{l+1}=\mathrm{MLP}(\mathrm{LN}({\stackrel{\mathrm{\Lambda }}{Y}}^{l+1}))+{\stackrel{\mathrm{\Lambda }}{Y}}^{l+1}.$$

3.2.

Cascaded Feature Enhancement Module Based on ASPP

Multiscale information plays a crucial role in improving the detection accuracy of clouds. General multiscale feature extraction architectures often lack effective information interaction among different scales, which can negatively impact detection performance. To address these challenges and effectively extract multiscale information, we propose the CFEM. It consists of two components: the multi-scale feature extraction module (MFC) and channel weight selection (CWS). The overall CFEM is given by Eq. (5)

Fig. 6

Architecture of CFEM.

The MFC component includes a pooling block, a $1\times 1$ convolution, and three parallel deformable convolution (DCN)³¹ modules. The pooling block takes the input features and applies both max pooling and average pooling operations. The results of these pooling operations are then element-wise added together. The resulting feature map is then upsampled using linear interpolation to restore it to the original size. DCNs use different dilation rates to capture cloud information at different scales. Compared with regular convolutions, the DCN is better suited to adapt to the shape of clouds and extract their semantic information. The regular convolution has a dilation rate of 1 and padding set to 0, whereas DCN has dilation rates and padding set to 6, 12, and 18. The kernel size of the regular convolution is $1\times 1$, with a stride of 1. All DCNs have a convolution kernel size of $3\times 3$ and a stride of 1. To enhance the cloud semantic information, we respectively concatenate $f_1$, $f_2$, and $f_3$with the input features $x$. The MFC module is defined as shown in Eq. (6)

CWS component includes a $3\times 3$ DCN, average pool, and MLP. The channel concatenation in the MFC module unavoidably leads to redundant information, which can negatively impact detection performance. CWS is utilized to concentrate on the significant channel information related to clouds and to diminish information redundancy. The specific equation of CWS is shown in Eq. (8)

3.3.

Feature Fusion Module Based on Attention Mechanism

In Secs. 3.1 and 3.2, the global contextual semantic information, local texture detail information, and multiscale information from cloud images are extracted by FEM and CFEM. However, the information contains some irrelevant noise and background information. We designed the FFM to fuse the aforementioned information and focus on foreground (cloud) information. The FFM consists of the spatial attention module (SAM) and multi-scale channel attention module (MCAM). SAM and MCAM operate on the outputs of the CNN and Swin branches, respectively. SAM is responsible for focusing on the spatial information and local texture details of clouds in the input information. MCAM extracts multiscale information from the intermediate layers and focuses on important channel information and spatial information relevant to clouds. The final output is obtained by merging the 1 × 1 convolution of the outputs from SAM and MCAM. The specific structure of the FFM is shown in Fig. 7 and Eq. (9)

Fig. 7

Architecture of FFM.

The SAM module applies max pooling and average pooling operations to the input features to extract various spatial features. Subsequently, the concatenated results undergo a $1\times 1$ convolution and sigmoid activation function to obtain the spatial attention weight matrix. The specific equation of SAM is shown in Eq. (10)

The MCAM module passes through three parallel convolutional modules for feature extraction at different scales. The convolutional kernel sizes are respectively set to $3\times 3$, $5\times 5$, and $7\times 7$, with a stride of 1. Then, the features at multiple scales are added together to obtain the fused feature. The fused feature is then subjected to average pooling and passed through an MLP to obtain a weight vector. Next, the weight vector is multiplied element-wise with the features extracted by the three convolutions. Finally, the fused feature undergoes the SAM to achieve the final result. The specific equation of MCAM is shown in Eq. (11)

3.4.

Loss Function

The binary cross-entropy (BCE) provides pixel-level loss supervision, ensuring accurate classification of individual pixels. The intersection over union (IOU) loss provides image-level loss supervision, reinforcing cloud pixels that were missed at the pixel level and weakening falsely detected cloud pixels. Therefore, we utilize the BCE loss in combination with the IOU loss to form our loss function for cloud detection, distinguishing between cloud and non-cloud. The equation of BCE loss is defined as Eq. (12)

3.5.

Evaluation Metrics

To evaluate the effectiveness of our model, we utilize metrics such as accuracy, precision, recall, and $F1$. The following are the equations for the evaluation metrics

5.1.

Comparison Methods

To validate the effectiveness of the model, this study compares STCNet with BoundaryNet,¹⁷ PspNet,³² CDNetv2,³³ RsNet,³⁴ and ManfanNet.³⁵ BoundaryNet is a model specifically designed for accurate cloud detection, aiming to address issues of unclear cloud boundaries and low accuracy. CDNetv2 is a cloud detection model that employs attention mechanisms and multi-scale feature fusion to achieve high-precision cloud detection on satellite image thumbnails. PspNet is a neural network model used for semantic segmentation tasks, which utilizes multi-scale fusion to combine features from different levels for more comprehensive and accurate semantic information. RsNet is a cloud detection model based on UNet,³⁶ which effectively integrates semantic information from deep and shallow layers through skip connections, improving the effectiveness of cloud detection. MafanNet enhances the ability to extract features and deeply mine spatial information through its designed attention modules, boundary refinement enhancement model, and bilateral FFM, thereby improving the detailed repair of cloud and cloud shadow boundaries.

5.2.

Comparison Experiments

All experiments were conducted on the GF1-WHU dataset. The results of different methods are shown in Fig. 8 and Table 1. In regions covered by thick clouds, as shown in Figs. 8(a) and 8(b), different models exhibit similar detection performance, except for differences in texture details. RsNet and PSPNet suffer from detail loss and edge smearing. MafanNet suffers from the adhesion of segmentation areas. CDNetv2 and BoundaryNet experience varying degrees of thin cloud and detail information loss. STCNet performs the best, reducing cloud omissions and producing clearer edges compared with other models.

Fig. 8

Comparison of three scene images using different methods. The types of three scenes are as follows: panels (a) and (b) represent scenes with thick clouds, panels (c) and (d) represent scenes with thin clouds, and panels (e) and (f) represent scenes with small clouds.

Table 1

Accuracy, precision, recall, and F1 scores for each testing method in the thick, thin, and small cloud regions.

In regions covered by thin clouds, as depicted in Figs. 8(c) and 8(d), DB-Net outperforms other models in detecting thin clouds and exhibits fewer instances of mistaking the background for clouds. BoundaryNet shows numerous cloud omissions, whereas CDNet and PSPNet not only have cloud omissions but also suffer from false cloud detections.

In addition, in regions covered by small clouds, as shown in Figs. 8(e) and 8(f), DB-Net exhibits better segmentation results and closer alignment with the ground truth in terms of details. BoundaryNet suffers from the loss of small clouds, whereas other models experience more severe omissions.

To further validate the effectiveness of DB-Net, we conducted quantitative analysis in different scenes. Table 1 presents the quantitative comparison results for thick, thin, and small cloud scenes. Figures 9 and 10 display the accuracy and $F1$_score for different scenes, respectively. In the three scenes, our model achieves the highest accuracy. In the thick cloud scene, the accuracy is 97.48%. In the thin cloud scene, the accuracy is 94.68%. In the small cloud scene, the accuracy is 96.55%. Compared with the highest accuracy, the accuracy improved by $\sim 0.53\%$, 2.7%, and 0.46% in the above scenes.

Fig. 9

Accuracy comparison for thick, thin, and small clouds under different models.

Fig. 10

F1_score comparison for thick, thin, and small clouds under different models.

Due to the distinct features of thick clouds and the reduced likelihood of foreground-background confusion, all metrics in the thick cloud scene are the highest among the three scenes. In the thick cloud scene, CDNetv2 achieves the highest precision, albeit at the cost of sacrificing recall. This trade-off reduces false positive detections of non-cloud pixels but also decreases the accurate detection of cloud pixels.

Compared with thick clouds, thin clouds have lower reflectance and less obvious features. Thin clouds also exhibit translucency and are easily confused with background features. These factors make it difficult to differentiate between foreground and background information. As a result, all metrics in the thin cloud scene are the lowest among the three scenes. In the thin cloud scene, RsNet achieves the highest recall, but this comes at the cost of increased detection of background information as clouds. ManfanNet achieves the highest precision rate, reducing the background information while also diminishing the effectiveness of cloud detection.

Small clouds occupy very few pixels and are easily overlooked during the feature extraction process. The metrics in the small cloud scene lie between those of the thin and thick cloud scenes. In the small cloud scene, although DB-Net has lower precision compared with CDNetv2, it achieves a balance between precision and recall, resulting in an overall improvement in the detection of small clouds.

Table 2 presents the quantitative evaluation results for all scenes, and it can be observed that the proposed DB-Net outperforms other methods in all metrics. DB-Net achieves an accuracy of 96.27%, a precision of 89.80%, a recall of 93.48%, and an F1 score of 91.60%. BoundaryNet exhibits a similar recall to DB-Net, but its performance in other metrics is significantly lower compared with DB-Net. DB-Net effectively suppresses non-cloud background information while preserving the information of thin and small clouds, achieving a balance between precision and recall. The metrics of other models are considerably lower compared with DB-Net.

Table 2

Comparative experimental results of different methods.

Overall, it can be observed that DB-Net performs the best in terms of results, effectively preserving small and thin cloud information while suppressing background information, thus achieving the best detection performance. BoundaryNet achieves detection performance that is close to DB-Net, but it has some shortcomings in terms of missing thin and small clouds. CDNetv2 and PspNet exhibit good detection performance in thick cloud scenes; however, they have significant instances of missed and false detections of clouds. MafanNet’s detection performance in thin cloud areas is second only to DBNet, but there is regional adhesion in thick cloud areas, and a large amount of detail is lost in small cloud areas. In contrast, RsNet performs the worst, exhibiting significant cloud omissions, false detections, and unclear edges.

5.3.

Ablation Experiments

To investigate the contributions of the three key modules in our proposed network, we conducted ablation experiments on the GF1-WHU dataset using Swin Transformer as the baseline. The results are shown in Table 3.

Table 3

Quantitative analysis results of the ablation experiments.

From Table 3, it is evident that different schemes result in improvements in model performance. Scheme 1 is based on baseline and incorporates the CFEM module, which aims to capture multi-scale information of clouds and focus on important channel information. Taking into account the local feature extraction capability of CNNs, we further add the CNN branch to scheme 1, forming scheme 2, which enhances local details and texture features. The FFM is added to scheme 2, forming scheme 3, known as DB-Net. DB-Net adaptively focuses on the semantic information of small and thin clouds while ignoring the background information of non-cloud objects.