Current image processing algorithm development is not based on an efficient mathematical structure that is specifically designed for image manipulation, feature extraction and analysis. Vast increases in image processing activities in such areas as robotics, medicine, and expert computer vision systems have resulted in an immense proliferation of different operations and architectures that all too often perform similar or identical tasks. Due to this ever-increasing diversity of image processing architectures and languages, several attempts have been made to develop a unified algebraic approach to image processing. However, these attempts have been only partially successful. In this paper, we define a heterogeneous algebra (in the sense of G. Birkhoff) which is capable of expressing all image-to-image transformations that can be defined in terms of finite algorithmic procedures. Conversely, for any image-to-image transformation defined as a finite sequence of terms in the image algebra, there is a structured program scheme that computes the transformation. Consequently, this algebra provides a common mathematical environment for image processing algorithm development, comparison, performance characterization, and optimization.