Many pan-sharpening techniques have been developed to synthesize a multispectral (MS) image at high resolution by fusing MS images and panchromatic (Pan) images. Most existing pan-sharpening methods can achieve results with high spatial resolution, but the spectral distortion in the fused results is still a problem that needs to be solved. In this paper, an adaptive linear model is proposed to reduce the spectral distortion by weakening the dependence on the correlation between Pan and MS. The difference between a Pan image and the combination of MS images is estimated by least square optimization, and embedded into the proposed model as a virtual band. According to the adaptive model, an iterative pan-sharpening algorithm is proposed based on the steepest descent method, in which the virtual band is used as a local adaptive constraint to the optimized solution. The proposed method is tested on datasets acquired by IKONOS, QuickBird, and Landsat 7 ETM+ and compared with the existing methods. The quality measures and the visual impressions show that the proposed method is an efficient approach to preserving spectral information and represents strong robustness against various scenes and sensors.