It is very critical that make full use of the local information for infrared dim and small target tracking. In this paper, an effective and fast algorithm based on the context learning is proposed to track infrared dim moving target. Firstly, the principle of the spatio-temporal context learning algorithm is described and the tracking deviation is analyzed. Then, a correlation filter is utilized to get a rational context prior for the dim moving target, the advantage is that the prior considers the image intensity information between a target and its surround pixels. Furthermore, a Gaussian high-pass filter is introduced to extract an accurate spatial context, which has little influence caused by the cluttered background. At last, the tracking problem is posed by computing a confidence map which takes into account sufficient information of a dim target and its surround background. Since the proposed algorithm is realized using fast Fourier transform, it is easy to be real-time. The experiments on various clutter background sequences have validated the tracking capability of the proposed method. The experimental results show that the proposed method can provide a higher accuracy and speed than several classical algorithms, including the improved Template Matching algorithm, the Temporal-Spatial Fusion Filtering algorithm and the Moving Pipeline Filtering algorithm.