Background: Defect compensation is one of the enabling techniques for high-volume manufacturing using extreme ultraviolet lithography. Aim: The advanced evolution strategy algorithm based on covariance matrix adaption is applied to compensation optimization to improve the convergence efficiency and algorithm operability. Approach: The advanced algorithm optimizes the solution population by sampling from the self-adapted covariance matrix of mutation distribution. Results: Optimization simulations for three different masks validated the algorithm’s advantage in convergence efficiency and searching ability compared with original differential evolution, evolution strategy, genetic algorithm (GA), and Nelder–Mead simplex method. The advanced algorithm employs fewer user-defined parameters and is proved to be robust to variations of these parameters. Conclusions: The advanced algorithm obtains better results compared with GA for best-focus, through-focus, and complex-pattern optimizations. With the inherent invariance property, appropriate operability, and robustness, we recommend applying this algorithm to other lithography optimization problems.