In this paper, we propose a novel method, called Evolution-based Outlier Removal (EOR) method, to remove outliers for robust geometric model fitting. We first select some data points and guide them to evolve towards the inliers. And then, we statistically analyze the evolutional results and distinguish inliers from outliers. Our main contribution in this paper is that, we develop a fitness function to improve the “quality” of selected point sets, which is then used to remove outliers. Experiments on real images illustrate the superiority of the proposed method over several state-of-the-art outlier removal methods.