Measurement of the roughness of solid surfaces by digital speckle correlation of video signals is reported. Speckle patterns appearing in the diffraction field of a laser-illuminated sample are taken by a CCD before and after the change of the incident angle or the laser wavelength, and their cross-correlation is calculated from which surface roughness can be evaluated. The theoretical cross-correlation function is first derived that describes speckle displacement and decorrelation. Its peak height is then provided as a function of the change of the incident angle or the wavelength. The decorrelation curve against speckle displacement is shown to depend on the surface roughness. The derived analytical relationships were verified by numerical simulations that used two-dimensional arrays of random numbers with various maximum values representing the maximum surface roughness. The relationship between the normal correlation calculated by the theory and the phase-only correlation adopted in the experiment was also evaluated by the simulation. In experiments various roughness standards for metal were measured by using a quick processor employing the phase-only correlation algorithm. The results were compared with those obtained from a mechanical stylus, and good agreement was observed between them for root-mean-square roughnesses between several micrometers and several tens of micrometers.