A single-channel color image encryption method is proposed based on iterative phase iterative process in quaternion Fourier transform domain. First, three components of the plain color image is confused respectively by using cat map. Second, the confused components are combined into a pure quaternion image, which is encode to the phase only function by using an iterative phase retrieval process. Finally, the phase only function is encrypted into the gray scale ciphertext with stationary white noise distribution based on the chaotic diffusion, which has camouflage property to some extent. The corresponding plain color image can be recovered from the ciphertext only with correct keys in the decryption process. Simulation results verify the feasibility and effectiveness of the proposed method.