We present an approach of color image interpretation and a recall scheme. The components of the system include image feature extraction, indexing process, and linguistic inference rules constructed for image interpretation. Based on these processes, each one of the segmented regions in an image can be described in terms of global linguistic meaning. We use the properties of color, texture, and spatial relationships to represent image features of an object. Each object can be first classified into a cluster by means of human experiences and knowledge, and can obtain the corresponding features of an object by using our proposed method. According to human experiences and cognition ability, linguistic inference rules can be further constructed. On the other hand, it can explain the interesting regions in an image. One of the distinguishing aspects of this system is the automatic recall process that uses both the localized region information and indexing process. The main procedure has two parts, which consist of the forward and recall processes. The forward process mainly achieves the linguistic meaning description of objects for an image. The recall process reconstructs the image region, which presents the rough mental image of human memory by means of the former result. Experiments show that this approach is reasonable and feasible.