The task of small target detection is to extract the small targets from the background image including clutter, noise and jitter background, so it is difficult to deal with. In this paper, after analyzing infrared small targets, noise and clutter model, we use a small window median filter to estimate the infrared background. Then using background cancelling method, that is, subtracting the estimated background from the source image, the resident image can be obtained. Finally, an adaptive threshold is used to segment the residual image to obtain the potential targets. Considering the computational load, the two-dimensional filter is simplified into a one-dimensional filter. Experimental results show that the algorithm achieved good performance and satisfy the requirement of real-time processing of large size infrared image.