As one of the eight wonders in the world, the virtual restoration of Terracotta Warriors is of great significance to archaeology. However, some parts of fragments were corroded for thousands of years, resulting in the existence of several holes in most of the restored cultural heritage artifacts. Based on the structural and textural information, we present a framework for filling the hole. First, a method based on the Poisson equation was employed to fill the hole on the triangular mesh model. Then, to complete the surface color and texture information of the three-dimension (3D) model, make the hole patch, and the original model surface texture natural transition, the 3D problem is converted into two-dimension (2D) image inpaint problem, and a refined network is added into EdgeConnect to generate a higher resolution result. A set of experiments is performed to evaluate the performance of our proposed framework. We hope the proposed framework can provide a useful tool to guide the virtual restoration of other cultural heritage artifacts.