Variable block size motion compensation has been adopted by the emerging video coding standard H.264. It can represent the motion characteristic in a macroblock more accurately, and therefore reduce the prediction error to achieve high compression gains. On the other hand, it causes high computational complexity in motion estimation at the encoder. The motion estimation exhaustively performed over all modes to find the best mode for inter coding is slow and computationally involved. In order to reduce the complexity, we proposed a fast inter mode decision algorithm for H.264 encoder. The proposed method efficiently determines a suitable block mode according to the motion field distribution and correlation within a macroblock. The experimental results show that the proposed method reduces a considerable amount of complexity at encoder, while the R-D performance of the proposed method is comparable to that of full mode search.