The emerging H.264 video coding Standard can achieve a substantial
coding gain as compared with existing coding standards. One major contribution of its gain comes from a very rich syntax for motion compensated prediction at the expense of a higher computational complexity. To be more specific, seven modes of different block
sizes and shapes (i.e. 16x16, 16x8, 8x16, 8x8, 8x4, 4x8 and 4x4) are supported. To do full search over all modes requires an extremely large amount of computation. We propose a fast search algorithm for the variable block size motion estimation. The proposed algorithm includes three stages. First, an initial estimate of the motion vector is obtained by a multi-resolution motion search. Then, based on the estimated motion vector and its corresponding distortion measure, a rate-distortion model is used to select the initial mode for motion search. Finally, some early-termination rules are adopted to reject impossible block sizes and motion locatios sooner. By avoiding search through all block sizes, the amount of computation involved in the motion search can be substantially reduced. The proposed algorithm can achieve a speed-up factor up to 120 times when compared to the fastest full-search algorithm.