Relevance feedback is a powerful and widely used technique in content-based image retrieval systems. However, most relevance feedback approaches use only weighted feature sum of the feedback images to optimize the query for refining image similarity assessment. Such approaches do not work very well in most cases, especially when the user wants to express an ‘‘OR’’ relationship among the queries. In this paper, we propose three methods, weighted distance sum, minimal distance (MD), and minimal distance rank (MDR), to measure the similarity between images in database and the feedback images in query refinement. After experimental comparisons we propose a relevance feedback scheme using the MDR method and the MD method to describe the user’s multiple intentions. Based on this scheme, an image retrieval and semiautomatic annotation system, iFind [Y. Lu, C. H. Hu, X. Q. Zhu, H. J. Zhang, and Q. Yang, ‘‘A unified framework for semantics and feature based relevance feedback in image retrieval systems,’’ in Proc. 8th ACM International Conference on Multimedia, pp. 31–37 (2000) and X. Q. Zhu, W. Y. Liu, H. J. Zhang, and L. D. Wu, ‘‘An image retrieval and semi-automatic annotation scheme for large image databases on the web,’’ in Proc. SPIE 4311, 168–177 (2001)], which integrates query refinement and semantic information, is presented. Experiments show that the proposed methods can result in substantial improvement in retrieval accuracy and can be especially useful for retrieval or annotating large image databases.