Generally, the process of monocular depth map generation consists of two stages: structure from motion and multi-view stereo. The multi-view stereo relies on the accuracy of the estimated camera pose and the photo-consistency assumption. However, the current methods cannot tackle the multi-view matching problem well because of the dependence on the accurate camera pose as well as the matching uncertainty. In this paper, to handle these issues, a new sparse-to-dense diffusion framework is put forward. First, the scene information is reconstructed from SFM (Structure From Motion) and the sparse point cloud is available instead of the camera pose. Secondly, the sparse depth point is re-projected to every frame as the depth label. Finally, the depth label is spread to the remaining pixels through a diffusion process. In addition, the edge detectors are used to make the propagation better-regulated. Experimentally, the results show that the proposed framework can robustly generate the depth map from monocular videos.