An algorithm for methodically deriving rate-distortion points for transform coefficient selection schemes for images is presented. The idea is to iteratively generate a set of convex hulls from which a composite operational rate- distortion curve is derived. Although this approach can be used to generate optimal interior rate-distortion points, the complexity is high. A fast suboptimal approach is then proposed which is based upon a modified version of threshold selection. In the modified threshold selection algorithm, each transform block operates on a point along a non-convex rate-distortion curve which is generated from rank ordering of coefficients in the block. Simulations of this fast algorithm using finely quantized DCT coefficients from an image with separate coding of amplitudes and runlengths show that very good rate-distortion performance can be obtained. These simulations also suggest that the modified threshold selection curve tends to lie within the first few convex hulls generated from the composite shell method. The modified threshold selection algorithm provides a fast way for achieving good rate-distortion performance in transform coding systems.