A moving object detection algorithm with sparse motion field estimation, motion classification and pixel-wise segmentation is proposed. Firstly, sparse motion field is recovered by fast corner detection and tracking. The corners that belong to the same motion pattern are classified according to their motion consistency, then, the resulting corner group is used to reconstructed scene image, and the foreground corners are identified by getting rid of the group with the least reconstruction error. Finally, optimal dense segmentation of the foreground is performed by using graph cuts, the energy function of which integrates corner motion, local color distribution and image edges. The proposed method is tested on the dataset of real complex scenarios and its effectiveness is demonstrated in the results.