We consider in this paper the problem of image inpainting in medical image analysis, where the objective is to reconstruct missing or deteriorated parts of an image. It is a good tool for such medical applications as vascular reconstruction, specular reflection removal for endoscopic images, MRA artefacts removing etc. Most inpainting approaches require a good image model to infer the unknown pixels. The proposed approach uses the modified exemplar-based technique. A novel approach combines mapping from image patches and pre-trained deep neural network. In our work, we exploit the concept of sparse representation, which takes a group of nonlocal patches with similar textures as the basic unit instead of a patch. Moreover, the color and multi-direction constraints are incorporated into the optimisation criterion to obtain sharp inpainting results. As a result, the proposed method provides plausible restoration while propagating information of edge for the target region. Experimental results demonstrate the effectiveness of the proposed method in the tasks of medical image inpainting.