Polarization sensor is a device to perceive the polarization information of skylight in polarization navigation. And imaging polarization sensor is better than the multi-channel point sensor for the advantages of wide field, large data redundancy and strong robustness. But the polarization patterns of skylight in different spectrums are different, which would represent different symmetry and structural similarity to the Rayleigh model. Meanwhile, the detectors inside have different responses in different spectrums. Therefore, the spectrum selection in the process of designing an imaging polarization sensor is an important issue. In this paper, we quantitatively evaluate the spectrum for imaging polarization sensor under the different sky conditions by two indexes: structural similarity (SSIM) index and symmetry error index. The result shows that wavelength shorter than 514nm has both higher structural similarity and symmetry in the visible spectrum. In other words, the imaging polarization sensor works better in the short-band spectrum.