Airborne light detection and ranging (LIDAR) data filtering is the most time-consuming and expensive part in applications related to laser scanning. This paper proposed a fast facet-based LIDAR data filtering method. LIDAR point clouds are interpolated onto a regular grid, and the filtering of nonground points is implemented on the grid-based data. The simple, quadratic, and cubic facet models, which are respectively, based on the zero, second, and third orders of orthogonal polynomials, are used to estimate the underlying elevation surface trend, which is considered as the approximation of ground surface. As the ground measurements are generally below the objects, the nonground points are filtered by removing the points that are higher than the estimated elevation surface trend. The resulting holes are filled with the nearest remaining measurements. Iteratively filtering in this way, the estimated elevation surface trend converges at the real ground surface. The nonground points that are higher than the finally approximated ground surface are filtered and the ground points are extracted from the LIDAR data. Experimental results on the test data released from the International Society for Photogrammetry and Remote Sensing (ISPRS) demonstrate that the proposed approach is efficient and provides at least comparable performance with the accuracy reports published by ISPRS.