Surface flatness assessment is required for controlling the quality of various products, such as building and mechanical
components. During such assessments, inspectors collect data capturing surface shape, and use it to identify flatness
defects, which are surface parts deviating from a reference plane by more than the tolerance. Laser scanners can deliver
accurate and dense 3D point clouds capturing detailed surface shape for flatness defect detection in minutes. However,
few studies explore algorithms for detecting surface flatness defects from dense point clouds, and provide quantitative
analysis of defect detection performance. This paper presents three surface-flatness-defect detection algorithms and our
experimental investigations for characterizing their performances. We created a test bed, which is composed of several
flat boards with defects of various sizes on them, and tested two scanners and three algorithms using it. The results are
reported in the form of a set of performance maps indicating under which conditions (using which scanner, scanning
distance, selected defect detection algorithm, and angular resolution of the scanner, etc.), what types of defects are
detected. Our analysis shows that scanning distance and angular resolution substantially influence the detection accuracy.
Comparative analyses of scanners and defect detection algorithms are also presented.