A simple and stable method is proposed for distinguishing dielectric and metal material surfaces from the polarization images captured by a vision system consisting of a linear polarizer and a digital camera. The polarization state is determined by the transmitted light intensity through the polarizer as a function of polarizing orientation. The degree of polarization (DOP) is estimated from the image intensities through the polarizer. The DOP map is quite effective for material classification around specular highlight on an object surface. We prove that the DOP map is convex for a dielectric surface and concave for a metal surface. The problem of material classification is then reduced to a simple judgment of the convexity of the DOP map obtained around the highlight peak. The proposed method is not a pixelwise local method based on thresholding the Fresnel ratio computed at each pixel but an area-based method based on the DOP map in a highlight area. The feasibility of the method is confirmed in experiments under a variety of conditions.