In classical compressive holography (CH), which based on the Gabor holography setup, two nonlinear terms are inherent in the intensity recorded by a 2D detector arrays, the DC term and the squared field term. The DC term (the term at the origin) can be eliminated by filtering the Fourier transform of the interference irradiance measurements using appropriate high-pass filter near the zero frequency. The nonlinearity caused by the squared field term can be neglected and modeled as a error term in the measurement. However, the above assumptions are significantly limited, which yields the degradation of reconstruction quality. In this paper, an novel scheme using phase-shifting method is presented. To accurately recover the complex optical field caused by the propagation of the object, without the influence of the DC term and the squared field term, a very effective method for removing these two terms is introduced. The complex optical field of the 3D object and the complex optical field at the detector plane can be precisely represented by a linear mapping model. The complex optical field at the recorder plane is obtained by phase-shifting interferometry with multiple shots. Then, the corresponded complex optical field at the detector plane can be successfully extracted from multiple captured holograms using conventional four phase-shifting interferometry. From such complex optical field at the record plane, including the amplitude and phase information, the complex optical field of the 3D object can be reconstructed via an optimization procedure. Numerical results demonstrate the effectiveness of our proposed method.