Building edge or boundary extraction is always one of the most important issues for earth observation, city planning, and other applications. However, for accurately extracting building edge, there are commonly two difficult challenges. Firstly, unwanted strong edges from road and other things can be hardly avoided to be recognized. Secondly, it is more serious that many low or very low contrast weak edges will be not detected. In order to deal with these two issues to a certain extent, in this paper, based on sparse SVM with dual-scale features, we propose a Building Edge Extraction method in a Dual-scale Classification way with Decision Fusion embedded (DC-BEE). Specifically, with global linearity information as priori knowledge, training samples are selected automatically at first. Next, a sparse SVM classifier is trained using the dual-scale local edge features of the training samples. And then, the trained sparse SVM is employed to classify all extracted edges. Finally, the dual-scale decision fusion strategy is performed for final building edge extraction. Visual analysis and quantitative analysis of the experimental results from different style city regions illustrated that the proposed DC-BEE method can efficiently fulfill the building edge extraction task automatically.