This paper proposes a robust detection and tracking method that can detect and track ship targets in cluttered forward looking infrared image sequences. The proposed method includes two major steps, target detection and target tracking. Because the target's imaging size will be scaled-up gradually along with the imaging distance being reduced, the target's imaging process is analyzed in detail and the target's imaging size is estimated in real-time. Based on the estimation of the target's imaging size, the targets with different imaging sizes can be treated as small targets by image scaling operation and detected successfully using the improved multilevel filter and iterative segmentation technique. In the tracking stage, the histogram back-projection technique, which has little effect on the movement of the camera, is applied to the mean shift procedure for real-time application. Extensive experiments demonstrate the utility and the performance of the proposed approach.