For the problem of ignoring color information and computing complexity and so on, a new target tracking algorithm based on improved SURF(Speed Up Robust Features) algorithm and Kalman filter fusion is studied. First, the color invariants are added in the generation process of SURF. And then the current position is predicted by using the Kalman filter and establishing the search window. Finally, the feature vectors in the search window are extracted by using the improved SURF algorithm for matching. The experiments prove that the algorithm can always track targets stably when the target appears scale changed, rotation and partial occlusion, and the tracking speed is greatly improved than that of the SURF algorithm.