We propose a novel segmentation algorithm called SMART for color, complex documents. It decomposes a document image into 'binarizable' and 'non-binarizable' components. The segmentation procedure includes color transformation, halftone texture suppression, subdivision of the image into 8 by 8 blocks, classification of the 8 by 8 blocks as 'active' or 'inactive', formation of macroblocks from the active blocks, and classification of the macroblocks as binarizable or non-binarizable. The classification processes involve the DCT coefficients and a histogram analysis. SMART is compared to three well-known segmentation algorithms: CRLA, RXYC, and SPACE. SMART can handle image components of various shapes, multiple backgrounds of different gray levels, different relative grayness of text to this background, tilted image components, and text of different gray levels. To compress the segmented image, we apply JPEG4 to the non-binarizable macroblocks and the Group 4 coding scheme to the binary image representing the binarizable macroblocks and to the bitmap storing the configuration of all macroblocks. Data about the representative gray values, the color information, and other descriptors of the binarizable macroblocks and the background regions are also sent to allow image reconstruction. The gain is using our compression algorithm over using JPEG for the whole image is significant. This gain increases as the proportion of the size of the subjects prefer the reconstructed images from our compression algorithm to those form the bitrate-matching JPEG images. In a series of test images, this document segmentation and compression system enables compression ratios two times to six times improved over standard methods.