When facing sea-sky background, island-shore background and other complex conditions, recognition rate and false alarm rate of the existing ship target recognition system based on a single wide-wave infrared image will be affected. For solving the above problems, this paper has studied the method of ship target recognition based on multi-spectral infrared images. The image data set of 5 medium-wave infrared images was collected and the samples data set was constructed by annotating the multi-spectral images manually. Firstly, Dense SIFT feature of each infrared image was extracted. Secondly, PCA was applied to each SIFT feature, reducing its dimensionality form 128 to 64. Then the spatial and spectral position information of each SIFT feature was integrated into the feature vector. Based on the Gaussian mixture model, the feature vectors of the multi-spectral images were encoded to obtain the Fisher vector representing the target. Finally, the linear SVM classifier was used to identify Fisher vector and further to recognize the target. Experimental results show that compared with single spectral infrared image, the recognition rate based on multi-spectral infrared images is higher. The recognition rate of the proposed algorithm reaches 0.97. Research indicates this paper provides a new method for ship target recognition based on multi-spectral infrared images.