In this paper, a new model-based tracking algorithm is proposed for tracking rigid objects in 6 degrees of freedom. Only one calibrated camera is used in the approach which can handle the motion of objects with known geometry. Information in 2D images from the camera would conduct motion in 3D space. The useful image features are contour edges of object to be tracked. The matching process includes two aspects of: (1) feature extraction using local minimum energy and (2) global matching of known 3D models against the projected features. The algorithm is robust to change in lighting and background. The small motion hypothesis is used for fitting feature energy which is defined as the negative absolute value of the edge strength. An autoregressive AR(1) model is employed for detecting incorrect matches in terms of the feature energy. We have found a new invariance-based method to eliminate false matches caused by strong shadow or occlusion. The invariance is the ratio of trigonometric functions of the angles formed by a polygon. Both performance analysis and real object tracking show that the proposed algorithm is effective and robust.