A pinhole is usually used as the pupil in traditional optical scanning holography (OSH) method. Although such a structure is relatively simple, the in-focus sectional image may be degraded by out-of-focus haze because of its difficulty to be eliminated in sectional image reconstruction. In this paper, a random-phase pupil is employed in OSH system to reduce the impact of defocus image. It is proved by the experimental results that the defocus image trends to be more easily dispersed into speckle-like pattern, and then it can be removed by connected component method in the future. Analysis also focuses on the correlation coefficient between the original image and the reconstructed images under the conditions of adopting a pinhole or a random-phase pupil. From comparison, as the defocus distance increasing, one can find that the correlation coefficient of image by using a random-phase pupil is decreasing faster than using a pinhole pupil.