Speckle interferometry is a well established technique for the optical characterization of rough objects, with the quantification of deformations as one particular application of interest. Owing to its common path property, a speckle-shearing interferometer is often the natural choice as a setup. Like other speckle techniques, however, speckle-shearing interferometry suffers from the existence of phase singularities present in the speckle patterns. Phase singularities introduce ambiguities into the phase unwrapping process and make this evaluation step highly sophisticated. In this work, we attempt to reduce the number of phase singularities by physical means, i. e. by applying an incoherent averaging of multiple, mutually independent speckle intensities. The effect of the incoherent averaging on the number of phase singularities has been investigated theoretically, by computer simulations, and experimentally. To obtain high contrast fringes in connection with a shearing setup, which would not be the case for a simple extended light source, a periodically structured light source with a period matched to the shear distance is applied. It turns out that the number of phase singularities may indeed be reduced, but only to a certain extent.