In speckle interferometry (SI), there are different techniques for acquiring information about a surface under test. The surface roughness is one parameter of interest. In the past, there have been different attempts to create models for relating the surface roughness to SI. By using, e.g., angular or spectral speckle correlation techniques, it is possible to estimate the roughness of a test sample qualitatively and to classify different magnitudes of roughness. Yet, there is no method to measure roughness quantitatively by using speckle interferometry. In this work, one possible reason for this restriction is investigated. By using a numerical simulation, it is shown that SI is not only sensitive with respect to a statistical distribution of a test sample but also to the kind of structure and inner dependency of the surface. Finally, a conclusion for roughness measurement using SI in general is drawn and an outlook is given.