We propose a three-dimension (3-D) depth-accuracy enhancement framework of time-of-flight (ToF) depth camera. First, we propose a frame interpolation from unsegmented, unmeshed, and noisy ToF depth video, and a novel temporal interpolative filtering method. This method allows us to increase integration time for given fixed frame rate. Because of the lack of explicit point correspondences between textureless depth images, global rigid motion is estimated from sparse 3-D corner feature matchings between two reference depth frames. Given global rigid motion, a nonrigid iterative closest point detection is adopted to find dense 3-D depth point correspondence for accurate frame interpolation. A temporal interpolative filtering without additional frame memory is introduced to enhance the depth accuracy after the frame interpolation. Experimental results demonstrate the promising performance of accuracy enhancement producing real-time depth video.