The accuracy of two sets of feature points is significant to remote sensing image registration based on feature matching. This paper proposes a novel image registration method based on geometrical outlier removal. The purpose of this algorithm is to eliminate most outliers and preserve as much inliers as possible. We formulate the outlier elimination method into a mathematical model of optimization, the geometric relationship of feature points is the constraint, and derive a simple closed-form solution with linear time and linear space complexities. This algorithm is divided into three key steps. First two remote sensing images are registered by scale-invariant feature transform(SIFT) algorithm. The initial feature points are generated by this step. Then the mathematical model is built and the optimal solution is calculated based on the initial feature points. Last we compare two recent registration results based on the optimal solution, and determine if it is necessary to update the initial feature points and recalculate. The experiment results demonstrate the accuracy and robustness of the proposed algorithm.