In this paper, a novel target detection and tracking algorithm based on visual attention is proposed. Firstly, the algorithm extracts saliency map of the first frame by improved visual attention algorithm, then detects targets which moving very slowly or even close to stationary after eliminating the interference of background factors. Secondly, it makes the mean shift algorithm’s kernel fixed bandwidth to be a dynamically changing bandwidth, so it not only retains the feature of traditional mean shift algorithm and can accomplish real-time tracking, but also can reduce background interference. Thirdly, the target model is established based on the saliency map, so the model is described by a variety of features. Therefore, when the target’s single feature changes, as size or shape, it still can detect the target. Lastly, it uses the modified meanshift algorithm to track moving targets, which can reduce the probability of losing target. Experimental results show that this algorithm is applicable to image sequences of both infrared and visible light, and it has good tracking performance. What’s more, the algorithm provides the motion information of the moving targets, so it gives a possibility for accurate positioning.