The infrared (IR) target recognizing and tracking technique is widely applied to many fields such as industries, navigation, weapon controlling and guiding and so on. Its application in military field has become the research hotspot. The stability of target tracking is the most important in military applications. However, it is difficult to track the aerial target because of the complex background and noise interference, especially from long distance, which make tracking targets even harder. In this paper, a novel image tracking system is designed, which uses template matching algorithm combined with Kalman filter. Because of the noise in image, the presence of occlusion, and the deformation of tracked target, some tracking algorithms may fail. So it is the main idea in this paper to merge the advantages from the tracking algorithms, and track the target real time. The algorithm for weak small targets from the image is based on template matching algorithm. In order to overcome the problems related to the changes of unpredictable circumstance, Kalman filter tracking algorithm is used. For the disadvantage of template matching algorithm towards occasions in target tracking, such as target occlusion, drastic change of image intensity, the relevant solutions are proposed. In cases when the target is occluded or moves more than the operational limits of the tracking module, Kalman filter is used to predict the object location. Thus, automatic detection and tracking of target in real-time is achieved and the proposed method is more robust in target tracking. The results show that the algorithm can realize target tracking under complicated scenes. It also improves the tracking stability, capacity of anti-interfering and running efficiency.