To monitor urban areas using a synthetic aperture radar (SAR) sensor, we propose a symmetric analysis-based building signature extraction method. Instead of using separated algorithms, a unified framework is proposed to extract both layover and shadow areas. Since these two primitives usually exhibit long strip patterns in very-high-resolution SAR images, symmetry axes are first delineated. After that, local features are extracted from both symmetry and range direction to better distinguish different primitives. Then, these local radiometric features are used to identify different categories (layover, shadow, and background) via an efficient multiclass logistic regression classifier. To discriminate individual primitives, geometric information is adopted via an improved Ramer Douglas Peucker algorithm, which also simplifies the parameters for describing these primitives. To further enhance accuracy, combinatory analysis is implemented to exclude some false detections, and then shadow areas are extended via a local region growing method. The proposed approach is tested on a 0.75-m resolution airborne C band SAR image. The experiments are carried out under both small- and large-scale scenes, and the comparative results show our method has some advantages in low-contrast target detection and false-alarm elimination.