When extreme ultraviolet (EUV) mirror systems having several high-order aspheric surfaces are optimized, the configurations sometimes enter into highly unstable regions of the parameter space. Small changes of system parameters lead then to large changes in ray paths and optimization algorithms fail. A technique applicable for any rotationally symmetric optical system that keeps the configuration away from unstable regions during optimization is described. A finite-aberration quantity is computed for several rays, and its average change per surface is determined for all surfaces. For not too large values of these average changes, optimization remains stable. A design for EUV lithography is discussed.