Surfaces with specular, non-Lambertian reﬂectance are common in urban areas. Robot perception systems for applications in urban environments need to function eﬀectively in the presence of such materials; however, both passive and active 3-D perception systems have diﬃculties with them. In this paper, we develop an approach using a stereo pair of polarization cameras to improve passive 3-D perception of specular surfaces. We use a commercial stereo camera pair with rotatable polarization ﬁlters in front of each lens to capture images with multiple orientations of the polarization ﬁlter. From these images, we estimate the degree of linear polarization (DOLP) and the angle of polarization (AOP) at each pixel in at least one camera. The AOP constrains the corresponding surface normal in the scene to lie in the plane of the observed angle of polarization. We embody this constraint an energy functional for a regularization-based stereo vision algorithm. This paper describes the theory of polarization needed for this approach, describes the new stereo vision algorithm, and presents results on synthetic and real images to evaluate performance.