Cracks (craquelure) and paint losses are the main types of deterioration of master paintings as they are ageing. We explore the potential of deep-learning-based methods for virtual restoration of paintings focusing on crack detection and their digital inpainting. For the crack detection stage, we develop a model that combines the benefits of multimodal convolutional (MCN) and autoencoder neural networks based on U-Net. The proposed model, dubbed U-Net multimodal convolutional, proves to outperform both MCN and U-Net architectures as well as the benchmark machine learning models for multimodal crack detection, both visually and in terms of objective performance measures. The second stage in our virtual restoration framework, the digital crack inpainting, is an adaptive adversarial network. The obtained virtual restoration results show clear improvement in comparison with the reference methods in this domain. Also, we propose an original way of training an adversarial neural network, which allows us to apply it more successfully in practice. A series of experiments shows encouraging results compared to the current state-of-the-art and confirms the huge potential of deep learning in crack detection and virtual restoration of master paintings.