In this paper, an optical stylus displacement sensing principle has been studied, which serving a composite holographic optical element as an off-focus detection element. The mathematical models of optical measuring system have been established by means of Operator Algebra. The analysis results show that: the displacement of the tested surface is proportional to the image spot size on the detector. The sensibility of the sensor is obviously affected by the ratio of the equivalent focus length of the holographic optical element to the focus length of the objective lenses. To improve the measuring resolution and S/N ratio, so called off-plane detection method has been adopted. With which the output of the measuring results have a better linearity with the displacement of the tested surface. No less than 0.01 micrometers high resolution with linear measuring range of 5 micrometers have been obtained. Some measuring experiments are given.