Free-form optics have a wide range of applications since they can simplify the structure of an optical system, meanwhile significantly improving the system performance. Compared to optics with traditional profiles, optical testing for freeform surface is more difficult. Although a laser interferometer can reach the surface measurement precision in nanometer scale, it has problems of a limited measurement range, a complex system configuration and relatively high requirements of working conditions. Fringe reflection technique (FRT) is gradually becoming a powerful tool for free-form surface testing owing to its advantages of simple system structure, high measurement accuracy, large dynamic range, etc. However, multiple groups of fringe images are required to display in two orthogonal directions respectively during the FRT measurement to obtain the corresponding surface gradient information. It hinders the fast detection of free-form surface to a certain extent. In order to overcome the above shortcoming, a phase retrieval algorithm based on color-frequency encoding for FRT is proposed in this paper, which can achieve the absolute phase meanwhile reducing the required number of fringe images for measurement. Experimental results demonstrate the effectiveness of the proposed method.