Doppler echocardiography is valuable for the diagnosis and management of several cardiovascular diseases. Automated analysis of Doppler images can significantly assist in decreasing the known variability of manual measurements and the burdensome of manual delineation and calculation. We propose a novel and fully automated method to detect and analyze spectral Doppler waves used for assessment of diastolic function from mitral inflow [MV] (peak E and A wave velocity), mitral annulus [MA] (peak E' and A' wave velocity), and pulmonary pressure (peak tricuspid regurgitation [TR] velocity). We used the Faster R-CNN deep learning-based method for Doppler, ECG, and anatomical ROIs localization. We then used ECG to segment Doppler signals into individual beats followed by assessing the quality of these beats using density-based method and Structural Similarity Index (SSIM). To segment the spectral envelope for each beat, we used a novel combination of k-means clustering algorithm and Gradient Vector Flow (GVF) snake algorithm. We used 701 Doppler images, collected from 100 patients acquired in the Clinical Center at the National Institutes of Health, to evaluate the performance of the proposed method against expert manual peak velocity estimation. The experimental results demonstrate the efficiency and robustness of the proposed framework in estimating peak velocity, and thus making it a viable candidate for use in clinical settings.