A new optical metric, termed resist deformation factor (RDF), to represent deformation of three-dimensional (3D) resist profile has been introduced into a source and mask optimization (SMO) flow to mitigate defects caused by a reactive ion etching (RIE) process at the lithography stage. Under the low-k1 lithography conditions with both a highly-coherent source and a complicated mask, the 3D resist profile is subject to top-loss or bottom footing, resulting in hotspots and/or defects after the RIE process. In order to represent the 3D resist profile on a fast lithography simulation, a sliced latent image along resist depth direction is used to define RDF as the ratio of integrated optical intensities within the resist pattern to those around its surrounding area. Then the SMO flow incorporating the RDF into its cost function is implemented to determine both a source and a mask as the 3D resist profile is less likely to deform. The result of new SMO flow with RDF shows 30% improvement of resist top-loss.