Deep neural networks (DNN) have been widely used in many applications in the past few years. Their capabilities to mimic high-dimensional complex systems make them also attractive for the area of semiconductor engineering, including lithographic mask design. Recent progress of mask writing technologies, including emergent techniques such as multi-beam raster scan mask writers, has made it possible to produce curvilinear masks with essentially “any” shapes. The increased granularity of mask shapes brings enormous advantages and challenges to resolution enhancement techniques (RET) such as optical proximity correction (OPC), Inverse lithography technologies (ILT), and other advanced mask optimization tools. Attempts of replacing the conventional segment based OPC by the ILT and other advanced solutions for full chip mask tapeout have been around for over a decade. Extremely slow mask data total-turnaround time is one of the major blocks. Therefore, its applications have been limited to small clip based applications such as for scanner source optimization, mask optimization only used for hotspot fixing and hierarchical memory designs. In this paper we present a new technique to apply DNN in our newly developed GPU-accelerated mask optimization platform, which reduces the runtime significantly without sacrificing the accuracy and convergence. This new tool combines deep learning, GPU computing platform and advanced optimization algorithms, and provides a fast and accurate solution for mask optimization in the sub-10nm tech nodes.