Fringe projection technology is often used for three-dimensional (3D) measurement, but it is difficult to measure highlight surfaces. Polarization systems are usually used to remove highlight surfaces. Polarizing filters can be used to eliminate the highlights of the image, but they may also cause the image to be too dark and affect the measurement accuracy. Otherwise, to ensure measurement accuracy, the complexity of the operation of the polarization system will be increased. A method of polarization-based camera intensity response function for 3D measurement is proposed. The intensity response function of the camera under the polarization system is established. It can avoid the complicated polarized bidirectional reflectance distribution function model and directly and quantitatively calculate the required angle between the transmission axes of the two polarizing filters. Then it is combined with the image fusion algorithm to generate the optimal fringe pattern. Experimental results demonstrate that this method significantly eliminates the effects of highlights in the image. The fuzzy transition area between the black and white fringes is effectively reduced, and the edge information of the fringes is correctly restored. Moreover, the high signal-to-noise ratio and contrast of the image are retained when the polarization filters are added.