We present a small-amplitude, high-frequency vibration measurement system. This system is based on the reflective holographic grating in a crystal of bismuth silicon oxide without applying an external electric field. A quarter-wave plate is applied in the reference beam path, with a polarizer after the crystal, to fulfill the quadrature condition when no electric field is applied to the crystal. A reflection configuration is used to obtain a good overlapping of the interference beams, which increases the beam coupling. The factors that affect the diffraction efficiency, including the signal-to-reference-beam intensity ratio and the recording angle, has been investigated. The experimental results coincide with the theoretical results, and the optimal conditions are obtained. The results of comparisons of our system with the vibrometer TEMPO show that the nanometer vibrations of a piezoelectric transducer can be reliably detected.