Digital holography is a powerful tool for non-contact quantitative phase imaging. Off-axis configuration remains a popular choice among the digital holography systems due to its ability to separate the dc and cross-terms in the recorded hologram in Fourier spectral space.However, compensating the off-axis tilt of the reference wave is one of the open challenges in the off-axis digital holography.Deng et al. proposed an off-axis tilt compensation method based on hologram rotation [DENG et.al. Opt. Let., 2017]. The off-axis tilt is removed by subtracting the phase of the digital reference hologram obtained by rotating the original specimen’s hologram from the retrieved phase corresponding to the original hologram. Nonetheless, Deng’s method is extremely time consuming due to the computation of Fourier transform,inverse Fourier transform and phase unwrapping for many times. In this paper, we propose a simple algorithm to compensate the off-axis tilt . Firstly, apply Fourier transform to the original off-axis hologram, filter out the first-order spectrum by band filter, then determine directly the spectrum of digital reference hologram from the spectrum of the original hologram, then filter out the first-order spectrum from the spectrum of digital reference hologram, apply inverse Fourier transform to the two first-order spectra to obtain two complex fields, then retrieve directly the phase difference from the two complex fields using the direct phase difference algorithm, then unwrap the wrapped phase map by the phase unwrapping algorithm. Finally, simulations and experiments are conducted to prove the validity of the proposed method. The results are analyzed and compared with those of Deng’s method, demonstrating that our method not only can speed up by more than 50% the calculation time, but also can improve measurement accuracy.