Ion beam milling is an emerging advanced optical fabrication technology capable of deterministic figuring of optical surfaces. Much of the work in ion milling to data has emphasized figuring of glass-like materials, such as fused silica, which do not significantly roughen during ion milling. However, for ion milling to reach its full potential as an advanced optical fabrication technique it must be applicable to a broad range of materials to interest in optical fabrication including polycrystalline metals, semiconductors, and ceramics. In order to assess the feasibility of ion milling, the effect of ion dose on roughness evolution was investigated for a variety of materials including: silicon, germanium, sapphire, silicon carbide, fused silica, aluminum, and copper. Single crystal silicon, germanium and sapphire as well as polycrystalline CVD silicon carbide did not significantly roughen during ion milling. The roughness evolution of aluminum, copper and gold thin films were also studied; fine grained gold films were found to remain smooth during ion milling.