We recently reported that phase encoded samples produced with metallic components (e.g. gold nano-particles) can be distinguished by means of polarized light. Classification was carried out using data obtained from speckle distributions. Despite this approach is very successful, it cannot be used with codes made of materials that do not change the state of polarization of the illumination source. In the present communication we analyze the feasibility of using optical diffusers as polarizing phase encoders for optical security systems. Preliminary optical results seem to support our thesis.