Layered depth video (LDV) is a sparse representation of MVD, which is considered as a promising 3D video format for supporting 3D video services. This format consists of one full view and additional residual data that represents side views. However, the amount of residual data becomes larger when the distance between the central view and side views increases. To address this problem, a new inpainting-based residual data generation method is proposed in this paper. Then, the inpainting-induced artifacts is considered as new residual data and the residual data of two side views is merged into one buffer to further reduce the amount of data. On the other hand, the block wise alignment is used for higher coding efficiency. And in order to fit the shape or distribution of residual data, a new compression algorithm for coding residual data is proposed. The experiments show high compression efficiency of the proposed method. The proposed method allows reduction of required bitrate of at least 30% comparing to classical LDV method, while they have the similar quality of intermediate virtual view in the terminal’s display.