Due to the limited depth-of-focus of optical lenses in imaging camera, it is impossible to acquire an image with all parts of the scene in focus. To make up for this defect, fusing the images at different focus settings into one image is a potential approach and many fusion methods have been developed. However, the existing methods can hardly deal with the problem of image detail blur. In this paper, a novel multiscale geometrical analysis called the directional spectral graph wavelet transform (DSGWT) is proposed, which integrates the nonsubsampled directional filter bank with the traditional spectral graph wavelet transform. Through combines the feature of efficiently representing the image containing regular or irregular areas of the spectral graph wavelet transform with the ability of capturing the directional information of the directional filter bank, the DSGWT can better represent the structure of images. Given the feature of the DSGWT, it is introduced to multi-focus image fusion to overcome the above disadvantage. On the one hand, using the high frequency subbands of the source images are obtained by the DSGWT, the proposed method efficiently represents the source images. On the other hand, using morphological filter to process the sparse feature matrix obtained by sum-modified-Laplacian focus measure criterion, the proposed method generates the fused subbands by morphological filtering. Comparison experiments have been performed on different image sets, and the experimental results demonstrate that the proposed method does significantly improve the fusion performance compared to the existing fusion methods.