Computer-aided diagnosis (CAD) schemes of mammograms have been previously developed and tested. However, due to using “black-box” approaches with a large number of complicated features, radiologists have lower confidence to accept or consider CAD-cued results. In order to help solve this issue, this study aims to develop and evaluate a new CAD scheme that uses visual sensitive image features to classify between malignant and benign mammographic masses. A dataset of 301 masses detected on both craniocaudal (CC) and mediolateraloblique (MLO) view images was retrospectively assembled. Among them, 152 were malignant and 149 were benign. An iterative region-growing algorithm was applied to the special Gaussian-kernel filtered images to segment mass regions. Total 13 Image features were computed to mimic 5 categories of visually sensitive features that are commonly used by radiologists in classifying suspicious mammographic masses namely, mass size, shape factor, contrast, homogeneity and spiculation. We then selected one optimal feature in each of 5 feature categories by using a student t-test, and applied two logistic regression classifiers using either CC or MLO view images to distinguish between malignant and benign masses. Last, a fusion method of combining two classification scores was applied and tested. By applying a 10-fold cross-validation method, the area under receiver operating characteristic curves was 0.806±0.025. This study demonstrated a new approach to develop CAD scheme based on 5 visually sensitive image features. Combining with a “visual-aid” interface, CAD results are much more easily explainable to the observers and may increase their confidence to consider CAD-cued results.