The first impression of any building to a person is its exterior and decoration, and therefore the quality of decoration project shows the more important position in building project. A lot of projects produce quality problem because of the material color difference, which exists universally at the common project, and is often found at the high-grade decoration; therefore, how to grasp and control the color change of building materials, and carry out color quantification, it has the very important meaning. According to the color theory, a computer vision system used in color quantification
measurement is established, the standard illuminant A is selected as the light source. In order to realize the standardization of color evaluation, the mutual conversion between RGB and XYZ color space is studied, which is realized by the BP network. According to the colorimetry theory, the computer program is compiled in order to establish the software system, and realize the color quantitative appraisement in whole color gamut. LCH model is used at quantifying the color of building materials, and L*a*b* model is used at comparing the color change. If the wooden floor is selected and laid improperly during family fitment, it is easy to present "flower face". The color also arises greater discrepancy using the laths of same tree. We can give the laying scheme using the color quantification system; at the same time, the color difference problem laying stone materials is also studied in this paper, and the solution scheme has been given using this system.