We demonstrate a method of displacement measurement based on polarization hopping of laser with optical feedback. The measurement system is composed of a half-intracavity He-Ne laser, a quarter wave plate and an external feedback mirror. When the conditions of the second polarization flipping are satisfied, the frequency of intensity modulation can be doubled, and one polarization switching corresponds to displacement of external mirror. Meanwhile, the intensity transfer between two polarization states will come out, i.e., an increase of one polarization light intensity always accompanies a decrease of another polarization light intensity. One period of intensity transfer can be divided into four domains: e-light, e and o-light, o-light, and no light, and each domain corresponds to change of external cavity length. According to appearing sequence of the four domains, the movement direction of external mirror can be distinguished. Our method can improve the resolution of displacement measurement 8 times that of conventional optical feedback, and reaches 40nm for a laser wavelength of 632.8nm.