1.

Introduction

Rapid earthquake damage mapping provides rapid, accurate, and comprehensive knowledge about the conditions of damaged area, which is vital in the disaster assessment and mitigation.^1–3 For decades, remote sensing techniques play an essential role in investigating damage information caused by earthquakes due to its prompt availability after disaster and wide coverage.^4,5 Especially, the increasing availability of very high-resolution (VHR) imagery has significantly contributed its wide use in the damage mapping fields.

The VHR remote sensing imagery offers the opportunity to detect the fine details for damage mapping. However, the rich spatial information presented in postearthquake VHR remote sensing images requires more sophisticated processing techniques, leading to considerable problems such as the increasing computation complexity.⁶ That is mainly because the postearthquake VHR remote sensing images generally exhibit a complex combination of various damage types. These various damage types show scale variance in a variety of damage structures and spatial layouts, which makes it challenging to obtain the classification maps of postearthquake VHR remote sensing images.⁷

Over the past few decades, considerable efforts have been made to improve the classification accuracy of postearthquake VHR remote sensing images. Intensive studies have focused on this topic based on handcrafted features elaborated from both spectral and spatial domains. Generally, the spectral features are mainly based on the brightness of each band, which are usually regarded as the primary features for recognition of targets objects.⁸ The application of the spectral features can be widely found in visual interpretation of damage objects, such as collapsed buildings and debris flow.⁸ Semantic spectral features with physical meanings have also been used as they can strengthen the reflectance discrepancy of different objects on specific wavelengths, such as the normalized difference vegetation index^9–11 and the modified normalized difference water index.¹² Image texture is further explored to demonstrate that the combination of texture and spectral features can give supplementary information for efficient damage mapping, such as the gray-level co-occurrence matrix.^7,13,14 Comparing with spectral properties, VHR imagery has a much richer spatial, which improves discriminative ability. For instance, Vu¹⁵ utilized morphological profiles to efficiently capture spatial information for rapid damage mapping in urban areas. Similarly, spatial filters (such as Canny filters and Gabor filters) were also proposed for the extraction of spatial features in the context of postearthquake VHR images.^2,16,17 However, these techniques rely on a prior fixed, albeit usually rich choice of a suitable data representation, which depends on the knowledge of the analyst and on the specificities of the image at hand. As a result, a few image classification methods in these studies can be practically used under time pressure.

Recently, deep learning,¹⁸ one of the state-of-the-art techniques in the field of machine learning and visual recognition, is identified as the best way to extract discriminative and representative high-level features.¹⁹ Deep learning can learn nonlinear spatial filters automatically and generalize a hierarchy of increasingly complex features.^20–22 A superiority of deep learning is that it learns features from the original data directly, showing great flexibility and capability than traditional classification methods.^18,20,22 Especially, convolutional neural network (CNN), constituted of stacked nonlinear adaptive layers, has been proved to be more efficient models in image processing. The entire system of CNN is trained end to end, from raw pixels to ultimate categories, thereby alleviating the requirement to manually design a suitable feature extractor.^23,24 This enables CNN to be widely utilized in hyperspectral image classification.^25–27 However, in spite of the remarkable achievement in the application of hyperspectral images, there still exist recurring limitations when applied in the feature extraction of high-resolution images.

One problem lies in the fixed receptive fields²⁰ of deep network, which make it unable to characterize the objects with varied size.²⁸ The fixed receptive field of deep CNN requires a fixed-size input. However, objects in remote sensing images often appear at various observation scales. The fixed-size input is unable to characterize the varied objects in images, which will lead to incomplete feature representation of the image contents. Consequently, the outputs of deep CNN suffer from speckles and noises. To tackle this problem, some recent studies implement the parallel CNN as multiscale CNN to extract different features and hence to produce complementary outputs. The parallel CNN combined the outputs of different individual CNN architectures with different-size inputs to achieve multiscale, convolutional representation of the natural objects.^20,29,30 However, this scheme still has many shortcomings. The parallel CNN method is equivalent to training multiple CNN structures in one algorithm, so both the number of samples required and the training time are significantly increased.

Another dilemma is that the improvement in representation capability of CNN has come partly at the price of reduced feature resolution.³¹ This is mainly caused by the repeatedly max-pooling and downsampling in CNN.³² The reduced resolution makes CNN insensitive to the object boundary, and thereby causes poor objects localization, even suffering a so called “salt-and-pepper” effect.^31,32 Previously, two ways are usually carried out to encounter the localization challenge. The first idea is to harness information from multiple layers in the convolutional network to conserve as much of the object boundaries information as possible.³³ The second strategy is to seek a superpixel representation, essentially delegating the localization task to a low-level segmentation method.³⁴

Labeling of samples is also a research challenge for CNN training.³⁵ This is mainly due to the fact that compared with traditional supervised methods, deep CNN-based methods usually need more training samples to overcome the overfitting problem.³⁶ The number of labeled datasets in practice is usually far from sufficient for a deep CNN architecture training. In view of scale variance of natural objects, the main important concerns for the samples labeling are related to the size and redundancy of the training set.³⁰ The size and quality of the training set have a direct impact on the execution time needed for training and on the final result of the classification.³⁰ The training set must, thus, be carefully chosen, avoiding redundancy patterns, but also ensuring a good representation of the considered classes.

To tackle these issues, multiscale segmentation approach is attractive.³⁷ It generates a set of segmentations of the same image at different scales, which allows that image objects are identified or extracted at several levels of segmentation detail.³⁸ The benefit is twofold. First, image segmentation has a potential to improve the localization accuracy. It preserves the abundant boundary information by partitioning a given image into a number of homogeneous regions.³⁹ Embedding the object boundary information of multiscale segmentations into the CNN can shape the real contour of geo-objects and filter out the spurious areas, thus potentially improving localization accuracy.⁴⁰ Furthermore, multiscale segmentation benefits for the sample selection. Image segmentation allows to extract both the objects and object contexts such as shape and texture based on the homogeneous regions in the segmentation results. In this way, samples can be directly selected from the multiscale segmentation results, where objects can be well-presented in their own observation scales.

In this paper, we presented a combined multiscale segmentation convolution neural network (CMSCNN) for rapid earthquake damage mapping. First, multiscale training samples database is constructed based on multiscale segmentation algorithm. Specifically, the mean shift (MS) algorithm is first conducted on the postearthquake VHR remote sensing images to derive regions at three different scales. Multiscale training samples for each class are directly selected from the three types of regions, and then resampled to the same size, which serve as CNN inputs. Then, CNN is trained based on the multiscale training samples database to exploit the complex features of damaged objects, which do benefit to generate the classification results with accurate identification of postearthquake geo-objects. Finally, to account for the boundary information of geo-objects with various sizes, an iteratively region-based max voting is conducted based on the multiscale segmentations derived from the first step, to generate the final multiscale classification maps. Several experiments were conducted in four postearthquake VHR images, and the results demonstrate that CMSCNN is effective for rapid high-resolution damage mapping and has substantial practical merit.

The rest of this article is structured as follows: Sec. 2 presents the proposed CMSCNN for postearthquake VHR image classification. Experimental results of the proposed method and the comparisons with other methods are reported in Sec. 3. Discussions of the experimental results are given in Sec. 4. Finally, some conclusions are drawn in Sec. 5.

2.

Proposed Method

In this paper, a CMSCNN method is proposed. As shown in Fig. 1, the proposed CMSCNN includes the following three steps: (1) multiscale training samples database construction; (2) preliminary CNN classification; and (3) combine multiscale segmentations and CNN classification results.

Fig. 1

Overall workflow of the proposed CMSCNN framework.

2.2.

Framework of Combined Multiscale Segmentation Convolutional Neural Network

It is noticeable that deep features extracted by the traditional CNN are generally robust and effective for complex image pattern descriptions, especially for the case of high-resolution damaged scenes. However, the deep features are extracted from receptive fields with fixed sizes by traditional CNN.^19,28 Geo-objects in postearthquake images are often observed at various scales, which implicates the traditional CNN with fixed receptive fields unsuitable for postearthquake images classification. Moreover, due to the subsampling processes of CNN, deep spatial-related features with high-level abstractions naturally fail to detect the edges and contours of complex objects. Hence, the output classification result of CNN is commonly coarse-resolution that is characterized by accurate identification but poor delineation of geo-objects. Accordingly, to improve classification accuracy, the CMSCNN is designed due to its ability to learn multiscale deep features and to keep clear edges of objects in classification results.

2.2.1.

Multiscale training samples database construction

To obtain deep feature representations, one priority is how to obtain enormous training samples for CNN. Generally, they are selected manually pixel by pixel. However, this method is time consuming and labor intensive, which is impractical for fast response to rapid earthquake damage mapping. In addition, due to the significant scale variance inherent in postearthquake objects, it is hard to select the accurate and homogeneous samples without any prior information. Multiscale segmentation algorithm can produce a set of homogeneous regions at different observation scales.⁴³ Therefore, in this paper, we utilize the regions produced by multiscale segmentation algorithm to generate training samples, which favors better generalization of the training samples.

Figure 2 shows the training samples by the use of multiscale segmentation algorithm. As shown in Fig. 2, the postearthquake VHR image is divided into a series of small regions at different scales using a segmentation algorithm. Then, for each class, the patches centered on the region are extracted as training samples for it. Moreover, considering the scale variance of different classes, the individual samples for each class are identified at the optimal level of segmentation details. In detail, samples of each class are first selected at three different sizes, and then all three types of samples are resampled to the smallest size. Finally, all training samples selected at their optimal scales form the training database.

Fig. 2

Multiscale training samples selection using multiscale segmentation algorithm.

The multiscale segmentation algorithm is significant for the construction of sample database. MS is a state-of-the-art segmentation algorithm with the advantages of simple parameter setting and no requirement for any prior knowledge.^2,44,45 Moreover, the ability of MS to maintain the saliency as well as the edge information has contributed to its wide applications in segmenting complex natural images, especially those without prior information. Therefore, we adopt MS here as the multiscale segmentation algorithm due to its outstanding ability of preserving the object boundaries.

2.2.2.

Convolutional neural network classification

In this paper, we employed a two-layer CNN framework to produce the preliminary classification maps. The framework is shown in Fig. 3.

Fig. 3

The general CNN framework. C1, S1, C2, and S2 represent the first and the second convolutional layer and subsampling layer, respectively.

As shown in Fig. 3, there are two convolutional layers followed by subsampling layers in the CNN. The CNN model is first trained using the multiscale training samples database. The parameters of this model empirically tuned as introduced in Sec. 3. Then, the trained CNN is used for images classification. Noteworthy, as for the inputs, for each pixel $p=(x,y)$ to be labeled, a patch of size $w\times w$ is extracted centered on $p$, which can take spatial information about the centered pixel under consideration.

2.2.3.

Combined multiscale segmentations

CNN predicts the presence and rough positions of target objects, but it has poor delineation for object borders, e.g., neglected spatial consistency. As a consequence, nonsharp boundaries and spurious regions will be generated, which tend to amplify the damage mapping uncertainty. For instance, an intact building in the collapsed ruins may appear as a speckle in the classification map. Multiscale segmentation algorithm is considered as a promising process to address this problem. The regions and region contexts information are well-preserved in the multiscale segmentation results. There is a possibility to combine multiscale segmentations and the coarse classification of CNN to improve the localization accuracy of objects. It is supposed that coarse-resolution classification of CNN is important to differentiate between different objects while fine resolution segmentations is necessary for localization.

Therefore, the classification of CNN is combined with multiscale segmentations by max voting to reach a more detailed classification result. The scheme refines the classification from fine to coarse iteratively. The combination starts with the fine segmentation to produce the classified image, and then the classified image is regarded as the new classification to be combined with the following segmentation. The reason behind this is to avoid the situation that coarse segmentation tends to neglect the small objects and the fine segmentation inclines to generate the spurious regions in geo-object. The scheme consists of three steps. First, the classification is mapped to each segmentation to assign classification labels to each pixel accordingly. Then, instead of using single pixels, region-based segmentation is regarded as the classification unit. In the voting process, CMSCNN randomly selects a region and predicts its label by assuming that the region belongs to the label that accounts for the majority of labeled values from all pixels within that region, whereas the other labels contained in the region are inferred as either noise or misclassified object boundaries. So, the label $S{C}_r$ of region $r$ is assorted as

Eq. (3)

$$S{C}_r=\arg {\max }_{t=1}^N\sum _i\sum _j\mathrm{sign}[f_{r(i,j)}=t],$$

where $f_{r(i,j)}$ is the label of the pixel $r(i,j)$ in region $r$ from the initial classification, and $(i,j)$ is the coordinates of the pixel $r(i,j)$. $N$ is the total number of expected classes. Finally, the combination is continued until all the segmentations are implemented.

In this way, the multiscale classification result confirms the accuracy of the CNN recognition, and refines different objects by the abundant multiscale information contained in segmentations. Thus, the CMSCNN learns the features of different levels, creating more robust classifiers.

3.

Experiments

3.2.

Experiments Setups

First of all, the segmentations were produced by MS to obtain the boundary information of objects at three different scales on these test images. Three scale parameters will affect the performance of the MS, named the window widths of color, spatial domain, and the minimum area size. For T1 in the first row, the window widths of color/spatial domain are set to 7/4, 8/6, and 9/4 pixels, respectively. As for T2 in the second row, the window widths of color/spatial domain are set to 7/6.5, 8/6.5, and 9/5.5 pixels, respectively. In terms of T3 in the third row, the window widths of color/spatial domain are set to 15/4, 20/4, and 24/5 pixels, respectively. The minimum area size is set to 20 pixels as default.⁴⁴ In addition, the training samples for each class are selected from the segmentations at three scales of $20\times 20$, $22\times 22$, $24\times 24$, and then downscaling them with a size of $20\times 20\text{pixels}$. And the patch of input data is set as $5\times 5$.The resultant segmentations were shown in Fig. 4, where the first to the fourth rows represented the test images T1 to T3, and the second to the fourth columns were the segmentations at three scales from fine to coarse, respectively.

Fig. 4

Three scale segmentation results of four test images. (a) The original postearthquake image: from top to down are T1 to T3. (b)–(d) The first to fourth rows are the three segmentations at different scales from fine to coarse for T1 to T3.

From Fig. 4, we can find that all the interesting boundaries are clearly present in the images. When different threshold values were applied to the original test images [e.g., Fig. 4(a)], multiscale segmentation results at different scales could be obtained corresponding to a level of the boundary details [e.g., Figs. 4(b)–4(d)].

The architecture of CMSCNN consists of two subsequent layers, as shown in Fig. 3. Each layer is composed of cascading structure with convolutional and polling stages. In terms of pooling stages, they are all set to be subsampling by a small factor of two to achieve a compromise between efficiency and accuracy. For the convolutional layers, we set the kernel size to 5, and set number of feature maps to 6 and 12 for two convolutional layers, respectively, by considering the size of training samples. Mini-batch strategy is adopted to update trainable parameters in the nets and the size of the training batch is set to 100 samples each. Learning rate is controlled as one and the number of training epochs is set to 200 to ensure the nets converges both quickly and accurately.

To verify the superiority of the proposed CMSCNN, two popular algorithms, including the support vector machine (SVM) and the conventional CNN, are adopted as the compared algorithms. SVM is advanced supervised kernel classification approach.⁴⁷ Specifically, the RBF kernel function was selected, and the parameter Gamma in kernel function was set to 0.015. To have a fair comparison, the classification results of SVM were also improved by the use of iteratively voting based on the same multiscale segmentations as CMSCNN. Therefore, we named the SVM here as combined multiscale segmentations SVM (CMSSVM). As for the conventional CNN, it is designed to demonstrate the superiority of the combined multiscale segmentations scheme of CMSCNN. Therefore, the training samples database and the parameters of CNN structure are all the same as CMSCNN. Moreover, the comparative evaluations of the three different algorithms are conducted qualitatively and quantitatively.

3.3.

Comparative Evaluations of Experimental Results

3.3.1.

Qualitative evaluation

The classification results are shown in Fig. 5, where Fig. 5(d) shows the ground-truth images of T1 to T3, Figs. 5(b) and 5(c) shows the classification results of T1 to T3 by conventional CNN and CMSSVM, respectively, and the results from CMSCNN are given in Fig. 5(c).

Fig. 5

The classification results of four test images by (a) CNN, (b) CMSSVM, (c) CMSCNN, and (d) the ground truths. From top to down are the results of T1 to T3 study areas.

Generally, as we can see from Fig. 5, the classification results indicate the superiority of the proposed CMSCNN compared with CNN and CMSSVM. Specifically, CNN classification results showed a salt-and-pepper appearance throughout study areas [Fig. 5(a)], whereas the classification maps of CMSCNN and CMSSVM were much more homogeneous [Figs. 5(b) and 5(c)]. However, as shown in Fig. 5(b), CNN has poor object delineation for intact buildings and the roads, resulting in the nonsharp or speckled regions, as shown in objects labeled in ellipses of T3 study area in Fig. 5(b). This is probably because that the convolution and max-pooling processes in CNN ignored the subtle details of objects. In addition, the CNN detected many spurious objects especially those with varied sizes and similar spectral characteristics, such as the landslides and the grassland (shown in red ellipses corresponding to T3 site in Fig. 5). This is mainly due to the fixed extractor of CNN, which limited the recognition of objects at different scales.

In contrast to the results of CNN, as shown in Fig. 5(c), the classification results of CMSCNN are more consistent and with satisfied structures. It is worth noting that compared with CNN, the classifications by CMSCNN show more realistic object shapes with merit of clear boundaries, especially the collapsed buildings areas with varied sizes [in yellow ellipses of T1 and T2 areas in Fig. 5(c)] and the landslides [in red ellipses of T3 in Fig. 5(c)]. It indicates the effectiveness of the combination strategy of CMSCNN, which combines CNN with multiscale segmentations.

As shown in Fig. 5(b), compared with the proposed CMSCNN, we can find that CMSSVM generally shows a similar appearance, where most object boundaries are correctly extracted, as it adopted the same scheme of combining multiscale segmentations as CNN. However, there are some misclassifications existing. For example, some isolated, thin, and elongated objects, such as roads, are confused with their surroundings, as shown in yellow rectangles of T2 site corresponding to Fig. 5(c). In addition, CMSSVM ignored some objects, such as the intact buildings in the yellow ellipses of T1 and T2 sites in Fig. 5(c). Further, CMSSVM also misclassified some damaged areas into the intact buildings. This discovery also indicates that, compared with traditional machine learning algorithms, the CNN can adopt more satisfied classification results for complex geo-objects.

3.3.2.

Quantitative evaluation

For quantitative assessment, the confusion matrixes are adopted to evaluate the performance of CMSCNN quantitatively. Table 1 summarizes the classification accuracy by three different methods for test images T1 to T3. For the accuracy of each class, the best obtained results were shown in boldface, and PA represents the product accuracy, UA represents the user accuracy, OA represents the overall accuracy, and $K$ represents the Kappa coefficient.

Table 1

The classification accuracy of T1 to T3 images using CNN, CMSSVM, and CMSCNN.

Method		CNN				CMSSVM				CMSCNN
Test images	/%	PA	UA	OA	K	PA	UA	OA	K	PA	UA	OA	K
T1	Forest	75.81	92.82	71.62	0.62	94.07	94.61	87.85	0.83	85.57	84.65	92.56	0.9
	Broken buildings	74.83	66.9			99.99	74.86			96.74	94.93
	Road	65.86	69.42			82.54	100			99.99	87.41
	Grassland	61.82	80.76			80.81	100			82.5	96.91
	Intact buildings	70.74	35.04			40.35	82.02			80.11	100
T2	Intact buildings	79.83	46.37	70.6	0.6	45.41	93.41	83.19	0.76	90.89	92.63	96.88	0.96
	Forest	63.14	96.71			87.83	92.53			95.94	99.98
	Broken building	69.69	61.08			86.97	71.06			97.93	90.83
	Road	67.68	80.41			69.79	79			100	100
	Grassland	83.96	25.97			58.6	45.47			88.27	99.96
	Debris flow	79.83	46.37			96.23	100			100	100
T3	Grassland	72.43	54.43	74.37	0.69	82.64	79.97	87.42	0.84	95.31	80.08	92.49	0.91
	Forest	60.43	87.49			74.6	100			74.6	100
	Debris flow	88.77	95.88			100	87.01			100	97.9
	Dammed lake	79.92	39.75			96.61	100			96.61	100
	Bare soil	62.65	43.53			66.08	70.23			97.15	92.15
	Landslides	83.23	97.14			89.68	96.88			91.58	100
	Concrete	67.42	92.34			93.92	83.13			85.68	80.21

As we can see from Table 1, CMSCNN has significantly improved the classification accuracy for all four test areas. For example, compared with CNN and CMSSVM, CMSCNN has increased the overall accuracy (OA) by 20.04% and 16.23%, as well as increased the kappa coefficient by 0.27 and 0.21, respectively. These quantitative indicators can also demonstrate the above findings. For example, from the classification maps, we can find that CMSSVM seriously confused the intact buildings into the broken building areas in T1, resulting in its small user accuracy of broken buildings in Table 1. In terms of T2 site, CMSCNN still performed best by the fact that OA was 96.88% and the Kappa coefficient was 0.96, significantly higher than those of CNN (70.60% and 0.60) and taking substantial advantage over those of CMSSVM (83.19% and 0.76). With respect to T3 shown in Table 1, CMSCNN again obtained the best results, which were slightly higher than CMSSVM but significantly outperformed CNN.

4.

Discussions

In this paper, we proposed a CMSCNN algorithm for the classification of postearthquake VHR remote sensing images. Due to the rich spatial information contained in high-resolution images and the complexity of the various damaged geo-objects, feature extraction is one of the biggest challenges for the analysis of postearthquake images. Traditional image classification methods require numerous image features to be empirically designed and depend on the knowledge of the analysts, which are time-consuming and often fail to achieve accurate interpretation of image. Recent machine learning methods are too limited to recognize the complex damage objects due to their shallow structures. As a result, few image classification methods in these studies can be practically used effectively and efficiently in postearthquake images. In this paper, the CNN, a well-known deep learning method, was chosen for automatic feature learning of postearthquake VHR images. With the hierarchical structure of the CNN, image features at higher levels can be automatically extracted. Moreover, CNN has shown satisfied robustness and accuracy in detecting complex targets. As the abstraction level increased, the extracted deep features demonstrated strong invariance in terms of semantic content. However, the method often fails to consider the scale variety of geo-objects due to its receptive fields with fixed sizes and fails to capture boundary information of objects due to the subsampling processes.

To this end, we combine the multiscale segmentations with preliminary CNN classification results for both efficient multiscale training sample selection and better extractions of targets’ boundary. As demonstrated in our experiments, the combination of image objects and deep features is quite effective. For one thing, it alleviates people from the time-consuming process of training sample selection and allows choosing more optimal training samples for each target class at various scales. For another, the combination of multiscale segmentations and deep learning method provides accurate targets’ localizations as well as identifications in the multiscale classified images. In addition, the final classification is capable to capture various targets due to the consideration of multiscale information.

5.

Conclusions

Rapid damage mapping has always been a fundamental but challenging issue in the field of damage assessment and emergency rescue. More accurate and efficient classification methods for the postearthquake high-resolution images are required. This paper presents a CMSCNN approach to combine deep CNN with multiscale segmentations, and demonstrated its usefulness in rapid damage mapping in postearthquake VHR images. The results showed that the methodology is able to accommodate the rapid damage mapping. This is contributed to the following three processes: (1) the selection of multiscale training samples, which is efficient and beneficial to the rapid formation of the training sample database of the damages encountered; (2) the simple structure (two layers) and a few training iterations (200 times) contributed to intense the workflow for the practical application; (3) the combination of multiscale segmentations and preliminary CNN classification results, which significantly improved the accuracy in both localization and classification in comparison to the conventional CNN and the CMSSVM technique. The quantitative and qualitative evaluations also validated the fact that such a scheme renders CMSCNN simple, practical, and appropriate for rapid high-resolution damage mapping.