The method of feature-based registration has been successful applied in registration of multi-source remote sensing images. Unfortunately, the mismatching still exists due to the complex textures, spectrum variation, nonlinear distortion and the large scale change. In this paper, we proposed a novel feature point matching method of multi-source remote sensing images. Firstly, the Fast-Hessian detector is to extract the feature points which are described by the SURF descriptor in the following step. After that, we analyze the local neighborhood structures of the feature points, and formulate point matching as an optimization problem to preserve local neighborhood structures. The shape context distances of the feature points are utilized to initialize matching probability matrix. Then relaxation labeling is adopted to update the probability matrix and refine the matching, which is aimed to maximize the value of the object function deduced based on preserving local neighborhood structures. Subsequently, the mismatching elimination method based on affine transformation and distance measurement is used to eliminate the residual mismatching points. During the abovementioned matching produce, the multi-resolution analysis method is adopted to decrease the scale difference between the multi-source remote sensing images. Also the mutual information method is utilized to match the feature points of the down sampling and the original images. The experimental results are shown that the proposed method was robust and efficient for registration of multi-source remote sensing images.