For the sake of polarimetric accuracy, polarization imaging systems based on liquid crystal modulators often work at one given wavelength due to the strong chromatic properties of the liquid crystal retarders. This often requires the use of narrowband filters which reduces the amount of light in the system and thus the signal-to-noise ratio. For applications where the main parameter of interest is the target/background discriminability rather than polarimetric accuracy, spectral filtering may not be the best option. In this work, we investigate the impact of broadening the spectrum of the light entering the system on the discriminability performance of passive and active polarimetric systems. Through simulations, we show that broadening the bandwidth of the illumination can increase the contrast between two regions, as the increase of light flux compensates for the loss of polarimetric precision. Moreover, we show that taking into account the chromatic characteristics of the components of the imaging system can further enhance the contrast. We validate these findings through experiments in passive and active configurations, and demonstrate that the illumination bandwidth can be seen as an additional parameter to optimize polarimetric imaging set-ups.