To improve the diagnostic accuracy of breast ultrasound classification, a novel computer-aided diagnosis system based on B-Mode ultrasound and color Doppler flow imaging is proposed. First, different feature extraction methods were used to obtain the texture and geometric features from B-Mode ultrasound images. In the color Doppler feature stage, both vascularity and hemodynamic features are studied by applying color distribution and periodograms analysis to Doppler signals. Finally, a support vector machine classifier with selected feature vectors is used to classify breast tumors into benign and malignant. The accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and areas under the ROC curves of the proposed system are 94.28%, 96.36%, 92.00%, 92.98%, 95.83%, and 0.9679, respectively. The experimental results demonstrate that the proposed computer-aided diagnosis system can improve the true-positive and decrease the false-positive diagnostic rate, which is useful for reducing the unnecessary biopsy and death rate.