Most existing video compression standards use block-matching motion estimation to exploit temporal correlation between frames. To reduce the costly computation of exhaustive searches on all possible motion displacements, researchers have developed many fast algorithms. Some of these successfully proposed algorithms take advantage of the motion correlations between adjacent macroblocks. On the basis of the idea of second-order motion correlation between macroblocks in particular, we developed a new algorithm that applies a set of adaptive search patterns to benefit the statistical model of motions. In addition, adaptive early termination rules are used to prevent the waste of unnecessary computation. Simulation results show that the proposed algorithm outperforms most other existing algorithms in the areas of speed and visual quality.