Ion beam figuring (IBF) provides a highly deterministic method for high-precision optical surface fabrication, whereas ion-induced microscopic morphology evolution would occur on surfaces. Consequently, the fabrication specification for surface smoothness must be seriously considered during the IBF process. In this work, low-energy ion nanopatterning of our frequently used optical material surfaces is investigated to discuss the manufacturability of an ultrasmooth surface. The research results indicate that ion beam sputtering (IBS) can directly smooth some amorphous or amorphizable material surfaces, such as fused silica, Si, and ULE® under appropriate processing conditions. However, for IBS of a Zerodur® surface, preferential sputtering together with curvature-dependent sputtering overcome ion-induced smoothing mechanisms, leading to the granular nanopatterns’ formation and the coarsening of the surface. Furthermore, the material property difference at microscopic scales and the continuous impurity incorporation would affect the ion beam smoothing of optical surfaces. Overall, IBS can be used as a promising technique for ultrasmooth surface fabrication, which strongly depends on processing conditions and material characters.