Video fire detection is playing an increasingly important role in our life. But recent research is often based on a traditional RGB color model used to analyze the flame, which may be not the optimal color space for fire recognition. It is worse when we research smoke simply using gray images instead of color ones. We clarify the importance of color information for fire detection. We present a fire discriminant color (FDC) model for flame or smoke recognition based on color images. The FDC models aim to unify fire color image representation and fire recognition task into one framework. With the definition of between-class scatter matrices and within-class scatter matrices of Fisher linear discriminant, the proposed models seek to obtain one color-space-transform matrix and a discriminate projection basis vector by maximizing the ratio of these two scatter matrices. First, an iterative basic algorithm is designed to get one-component color space transformed from RGB. Then, a general algorithm is extended to generate three-component color space for further improvement. Moreover, we propose a method for video fire detection based on the models using the kNN classifier. To evaluate the recognition performance, we create a database including flame, smoke, and nonfire images for training and testing. The test experiments show that the proposed model achieves a flame verification rate receiver operating characteristic (ROC I) of 97.5% at a false alarm rate (FAR) of 1.06% and a smoke verification rate (ROC II) of 91.5% at a FAR of 1.2%, and lots of fire video experiments demonstrate that our method reaches a high accuracy for fire recognition.