The Laser Doppler Vibrometry (LDV) can continuously measure the speed, acceleration and other parameters of the micro-vibration target. However, with the increase of the surface roughness and the decrease of the diffuse reflectance of the measured object, or the increase of the measurement distance, the laser echo energy will decrease rapidly according to the basic theory of optical propagation. The typical laser doppler vibration measurement technology is based on the principle of two-beam heterodyne interference. However, due to the limitation of the optical device, the third beam, namely the system stray light, from multiple optical devices in the optical path, is inevitable, and has a certain intensity and phase distribution. The system stray light will seriously nonlinear suppress the vibration phase signal of the micro-vibration target. To reduce the system stray light, methods like surface coating, tilt design and curved lens design are employed, but it greatly affects the detection signal-to-noise ratio and micro-vibration detection capability of the system. In this paper, a method based on four-wave hybrid interference to eliminate stray light is proposed. A calibration laser beam is added to the original interference optical path. With measurement on the system stray light, the laser external modulation is used to make the intensity, frequency and phase of the calibration beam vary with the parameters of the system stray light. The influence of stray light in the system can be theoretically eliminated if the intensity and frequency of the calibration beam is consistent with the stray light, and the initial phase difference is π.