Template matching is a basic algorithm for image processing, and real-time is a crucial requirement of object tracking. For real-time tracking, a fast template matching algorithm based on grey prediction is presented, where computation cost can be reduced dramatically by minimizing search range. First, location of the tracked object in the current image is estimated by Grey Model (GM). GM(1,1), which is the basic model of grey prediction, can use some known information to foretell the location. Second, the precise position of the object in the frame is computed by template matching. Herein, Sequential Similarity Detection Algorithm (SSDA) with a self-adaptive threshold is employed to obtain the matching position in the neighborhood of the predicted location. The role of threshold in SSDA is important, as a proper threshold can make template matching fast and accurate. Moreover, a practical weighted strategy is utilized to handle scale and rotation changes of the object, as well as illumination changes. The experimental results show the superior performance of the proposed algorithm over the conventional full-search method, especially in terms of executive time.