In this paper, we propose a methodology for cross matching color face images and Short Wave Infrared (SWIR) face images reliably and accurately. We first adopt a recently designed Boosted and Improved Local Gabor Pattern (ILGP) encoding and matching technique to encode face images in both visible and SWIR spectral bands. We then apply newly developed feature selection methods to prune irrelevant information in encoded data and to improve performance of the Boosted ILGP. The two newly developed feature selection methods are: (1) Genuine segment score-based thresholding and (2) AdaBoost inspired methods. We further compare the performance of the original Boosted ILGP face recognition method with the performance of the modified method that involves one of the proposed feature selection approaches. Under a general parameter set up, significant performance improvement is observed.