Image distortion problem is key issue in DMD digital lithography system, in this paper, quality optimization algorithm of digital lithography based on improved particle swarm optimization algorithm is proposed. The fidelity is adopted as the fitness function. The pixels in the mask pattern are used as particles, and then optimization is implemented by updating the velocities and positions of these particles. Two different graphs are used to verify the method, image quality optimization of the standard particle swarm optimization algorithm and the steepest descent gradient descent algorithm, the pattern errors are reduced by 95.48%, 91.95% and 92.78%, 87.28%, respectively. The quality of image is improved, and the convergence speed is faster.