In this paper, we propose a method for target detection and tracking in infrared video. The target is defined by its location and extent in a single frame. In the initialization process, we use an adaptive threshold to segment the target and then extract the fern feature and normalize it as a template. The detector uses the random forest and fern to detect the target in the infrared video. The random forest and fern is a random combination of 2bit Binary Pattern, which is robust to infrared targets with blurred and unknown contours. The tracker uses the gray-value weighted mean-Shift algorithm to track the infrared target which is always brighter than the background. And the tracker can track the deformed target efficiently and quickly. When the target disappears, the detector will redetect the target in the coming infrared image. Finally, we verify the algorithm on the real-time infrared target detection and tracking platform. The result shows that our algorithm performs better than TLD in terms of recall and runtime in infrared video.