The rate constrained block matching algorithm (RCBMA) jointly minimizes displaced frame difference (DFD) variance and entropy, or conditional entropy of motion vectors for determining the motion vectors. It is intended for use in low rate video coding applications, where the contribution of the motion vector rate to the overall coding rate might be significant. The DFD variance versus motion vector rate performance of RCBMA employing size K x K blocks is shown to be superior to that of the conventional minimum distortion block matching algorithm (MDBMA) employing size 2K x 2K blocks. Constraining of the entropy or conditional entropy of motion vectors in RCBMA results in smoother and more organized motion vector fields than those output by MDBMA. The motion vector rate of RCBMA can also be precisely controlled for each frame by adjusting a single parameter.