In phase-shifting digital holographic interferometry for measuring the displacement distribution of an object, holograms and reconstructed images have speckle noise, and they cause large errors in the calculation of the displacement analysis. In order to reduce the effect of speckle noise and hence to increase the sensitivity of the measurement of the displacement, we previously proposed a method using windowed holograms. In this paper, we propose a new method for averaging the obtained phase-difference values. Many phase-difference values at a point of the reconstructed image obtained with different windows for a hologram are averaged with weights. In order to check the effect on accuracy, the number n of windows used for averaging is varied. The weight, which is the m'th power of the absolute value of the complex amplitude of the reconstructed object, is also varied. As a result, when n becomes larger, the standard deviation of the errors becomes smaller. When the power m is 2, the error becomes a minimum. The standard deviation of the errors in the case of a flat plate with 316-nm out-of-plane displacement is 88 pm when n=1024 and m=2.