We propose an advanced motion-compensated prediction method for improving the coding efficiency in H.264/AVC. Nine separable two-dimensional interpolation filters are applied to precise compensation for motion in various directions. A new optimal cost function, which considers the bit rate and distortion for coding the macroblock, is also proposed. The filter is adaptively selected per macroblock to minimize the proposed cost function. Also, an algorithm for reducing the overhead of transmitting the filter coefficients is described. In experimental results on the various standard QCIF/CIF test video sequences, the proposed method shows improvement in coding efficiency over conventional methods. This leads to approximately a 7.30% (for one reference frame) and 4.46% (for five reference frames) bit-rate reduction on average, compared to H.264/AVC.