We propose a pairwise Rayleigh quotient (PRQ) classifier and apply it to discriminate between malignant tumors and benign masses in mammograms. The PRQ classifier employs a Rayleigh quotient based on a set of pairwise constraints, which leads to a generalized eigenvalue problem with low complexity of implementation. Kernel functions are used to incorporate nonlinearity. Studies were conducted with features of 57 breast masses, of which 20 are related to malignant tumors and 37 to benign masses. The linear PRQ classifier provided results comparable to those obtained with Fisher's linear discriminant analysis (FLDA), support vector machines (SVMs), and convex pairwise SVMs (CPSVMs). The linear PRQ classification performance of the comparatively weak feature sets with edge sharpness and texture features was significantly improved by about 5%, as compared to those obtained by FLDA, SVM, and CPSVM. The nonlinear PRQ classifier with the triangle kernel provided the perfect performance of 1.0 in terms of the area under the receiver operating characteristics curve, for nearly all feature combinations, but with good robustness limited to the kernel parameter in a certain range. We propose a measure of robustness to evaluate the PRQ classifier.