Eliminating the sign ambiguity is one of the important steps in phase extraction from a single interferogram. We propose a new method based on energy minimization to remove the sign ambiguity when retrieving a phase. Markov random field (MRF) is used to build up the pairwise relationship of a sign map. The method is thus a compromise between “global” and “local” methods. To alleviate the disturbance of noise, we employ additionally a neighborhood-direction-averaging and an adaptive filter instead of using optical flow. The gradient orientation is properly smoothed for MRF observation. Various techniques for minimizing MRF energy are compared and simulation results demonstrate the effectiveness of our method.