Deriving models for lithographic masks based either on first principles or using an empirical model is becoming increasingly challenging as complex effects (once relegated to noise level) become more relevant. Deep Learning offers an alternative solution that can leapfrog the shortcomings of these previous approaches but requires a source of input data that contains enough diversity to allow an effective training of the neural networks. The solution for mask lithography modeling presented in this paper makes use of carefully calibrated SEM images to extract the information required to allow the training and testing of a deep convolutional neural network that achieves accuracy beyond what can be done in metrology-based methods. We demonstrate how the input data is calibrated to be consumed in this flow and present examples demonstrating its predicting power which can, for instance, detect the location and shape of hotspots in the layout. One significant additional advantage is the improvement in the ease and speed of building models compared to previous solutions which can dovetail well with regular production flows and can be adapted to dynamic changes in the mask process.