The simplistic infrared target observation feature is the main reason for that the particle filter tracking algorithm are always failed. Considering the limitations of infrared target feature description, we constructed an observation model based on high frequency features and high frequency features. The template update method was also analyzed. Finally, we designed the flow of particle filter tracking algorithm combining the gray feature and the high frequency feature. The simulation results show that proposed method has the good robustness and accuracy in infrared target tracking under complex background.