We present a technique to reduce speckle in visible-light optical coherence tomography (vis-OCT) that preserves fine structural details and is robust against sample motion. Specifically, we locally modulate B-scans orthogonally to their axis of acquisition. Such modulation enables acquisition of uncorrelated speckle patterns from similar anatomical locations, which can be averaged to reduce speckle. To verify the effectiveness of speckle reduction, we performed in-vivo retinal imaging using modulated raster and circular scans in both mice and humans. We compared speckle-reduced vis-OCT images with the images acquired with unmodulated B-scans from the same anatomical locations. We compared contrast-to-noise ratio (CNR) and equivalent number of looks (ENL) to quantify the image quality enhancement. Speckle-reduced images showed up to a 2.35-dB improvement in CNR and up to a 3.1-fold improvement in ENL with more discernable anatomical features using eight modulated A-line averages at a 25-kHz A-line rate.