In this paper, we propose a robust tracking method for infrared object. We introduce the appearance model and the sparse representation in the framework of particle filter to achieve this goal. Representing every candidate image patch as a linear combination of bases in the subspace which is spanned by the target templates is the mechanism behind this method. The natural property, that if the candidate image patch is the target so the coefficient vector must be sparse, can ensure our algorithm successfully. Firstly, the target must be indicated manually in the first frame of the video, then construct the dictionary using the appearance model of the target templates. Secondly, the candidate image patches are selected in following frames and the sparse coefficient vectors of them are calculated via ℓ<sub>1</sub>-norm minimization algorithm. According to the sparse coefficient vectors the right candidates is determined as the target. Finally, the target templates update dynamically to cope with appearance change in the tracking process. This paper also addresses the problem of scale changing and the rotation of the target occurring in tracking. Theoretic analysis and experimental results show that the proposed algorithm is effective and robust.