Traditional displacement measurement systems by grating, which purely make use of fringe intensity to implement fringe count and subdivision, have rigid demands for signal quality and measurement condition, so they are not easy to realize measurement with nanometer precision. Displacement measurement with the dual-wavelength and single-grating design takes advantage of the single grating diffraction theory and the heterodyne interference theory, solving quite well the contradiction between large range and high precision in grating displacement measurement. To obtain nanometer resolution and nanometer precision, high-power subdivision of interference fringes must be realized accurately. A dynamic tracking down-conversion signal processing method based on the reference signal is proposed. Accordingly, a digital phase measurement module to realize high-power subdivision on field programmable gate array (FPGA) was designed, as well as a dynamic tracking down-conversion module using phase-locked loop (PLL). Experiments validated that a carrier signal after down-conversion can constantly maintain close to 100 kHz, and the phase-measurement resolution and phase precision are more than 0.05 and 0.2 deg, respectively. The displacement resolution and the displacement precision, corresponding to the phase results, are 0.139 and 0.556 nm, respectively.