Vehicle color recognition plays an important role in the intelligent transportation system. Most of the state-of-art methods roughly take all pixels into consideration, but many parts of cars such as car windows and wheels contain no color information. Also these methods do not work well enough in reducing the influence of sunlight. In this paper, we propose a novel approach that aims to estimate the RGB value of the car body rather than just classify the vehicle’s color and achieve state-of-art performance. We try to filter the useless parts automatically and estimate the influence of sunlight on each pixel by introducing the specular-free image and the weighted-light-influence image. Experimental results demonstrate the performance of the proposed scheme in differentiating cars with very similar color.