Image registration is a difficult problem when dealing with images captured by different sensors, for instance, Visible and IR sensors, due to their texture, gray not matched and feature inconsistent. Since the edges of the objects present in the images are preserved in most cases, so, in this paper, a new contour-based image registration algorithm is proposed. First, an edge detection method based on multi-scale and multi-direction morphology is proposed for extracting contour. Then the matched contour pairs are found according to the following shape attributes, such as the first and second invariant moments, etc. Next, within these matched contours, several control-point pairs are selected, which has the characteristics of 1) even distribution 2) a suitable number of points 3) locally maximum curvature. In addition, the sea-sky-line is extracted in advance, which determines the region of image registration and reduces the computation time. The performance of our algorithm is demonstrated by estimating the registration accuracy and evaluating the fusion effects of the visual and IR images of ship target.