In order to achieve the purpose of infrared object detection, two phases are essential, including feature selection and matching strategy. Good Features should be discriminative, robust and easy to compute. The matching strategy affects the accuracy and efficiency of matching. In the first stage, instead of the joint distribution of the image statistics, we use region covariance descriptor and calculate region covariance using integral images. The idea presented here is more general than the image sums or histograms, which were already published before. In the second feature matching stage, we describe a new and fast pyramid matching algorithm under the distance metric, which performed extremely rapidly than a brute force search. We represent an object with five covariance matrices of the image features computed inside the object region. Instead of brute force matching, we constructed the image pyramid and decomposed the source image and object image into several levels, which included different image resolutions. After the completion of coarse match, fine-match is essential. The performance of region covariance descriptor is superior to other methods, and the pyramid matching algorithm performs extremely rapidly and accurately, as it is shown, and the large rotations and illumination changes are also absorbed by the covariance matrix.