We present a new image segmentation algorithm based on Markov random fields (MRF) with adaptive neighborhood (AN) systems. First, a new criterion function is developed to adaptively select the neighborhood system of the MRF. Second, thanks to the appropriate format of the new criterion function, a general iterated conditional mode (GICM) is deduced to incorporate the AN selection into the inference process in the segmentation of images. Third, an end points detection step is introduced to preserve the ends of line structures in the images. The proposed algorithm has the following advantages over previous works: the convergence of the new algorithm is much faster than classical algorithms; the AN configuration of each site is iteratively optimized in the inference process, which leads to more accurate segmentation results; and no extra knowledge is needed in the AN selection. Numerical experiments on a wide range of images demonstrate that the proposed image segmentation algorithm performs better, in terms of speed and detail preserving, than the algorithms based on classical MRFs.