We describe a dwell time algorithm for the polishing of small axis-symmetrical aspherical surfaces. The dwell time distribution of the scanning polishing tool on the rotating workpiece is calculated to reduce the residual surface error. The dwell time at each discrete grid is calculated as an integer multiple of the workpiece rotation period, which is also useful for the spatially varying case in the local polishing area. A spherical polyurethane tool with abrasives is adopted for a computer- controlled polishing process. A linear algebraic equation of removal depth, removal matrix, and dwell time is derived by convolution of the removal depth at the dwell positions. The nonnegative least-squares method gives a solution to minimize residual error. Parametric effects such as the dwell grid interval are simulated. Finally, an experiment for tool mark removal is performed and the dwell time algorithm is evaluated to be valid.