Motion estimation (ME) is a computationally intensive task during video compression. Although diamond search (DS) is an efficient way to speed up block-matching process, skip mechanism (Skip-M) combined with DS will give further speed-up. However, how to choose threshold in this mechanism is essential to improvement magnitude. In this paper, we propose a scheme in which the threshold is chosen based on analysis of noise variance due to video devices. Simulation results show the proposed algorithm can remarkably reduce the time consumption with only marginal performance penalty.