Markov random field (MRF) model is an effective tool for polarimetric synthetic aperture radar (PolSAR) image classification. However, due to the lack of suitable contextual information in conventional MRF methods, there is usually a contradiction between edge preservation and region homogeneity in the classification result. To preserve edge details and obtain homogeneous regions simultaneously, an adaptive MRF framework is proposed based on a polarimetric sketch map. The polarimetric sketch map can provide the edge positions and edge directions in detail, which can guide the selection of neighborhood structures. Specifically, the polarimetric sketch map is extracted to partition a PolSAR image into structural and nonstructural parts, and then adaptive neighborhoods are learned for two parts. For structural areas, geometric weighted neighborhood structures are constructed to preserve image details. For nonstructural areas, the maximum homogeneous regions are obtained to improve the region homogeneity. Experiments are taken on both the simulated and real PolSAR data, and the experimental results illustrate that the proposed method can obtain better performance on both region homogeneity and edge preservation than the state-of-the-art methods.