A novel approach by the fusion of airborne laser scanning data and optical high-resolution imagery is proposed for automatically obtaining building boundaries with precise geometric position and details. The high-resolution images are used to directly extract the building boundaries with precise geometric position, and the laser scanning data are integrated to improve the correctness and completeness of the extracted boundaries. In this approach, a new method is first proposed to estimate the principal orientations of a building, based on the building image and rough principal direction constraints, which ensures the accuracy and robustness of the subsequent line segment extraction. On this basis, accurate boundary segments are extracted using a method based on laser scanning point density analysis and K-means clustering. Images from different sensors, including orthoimage, aerial stereo, or some other images, are able to be processed effectively. Experiments covering more than 200 buildings with various orientations, various structures, and various texture conditions are employed to test the proposed approach. The average correctness and completeness of the determined boundaries are 95% and 90%, respectively.