Optical spectroscopy is sensitive to morphological composition and has potential applications in intraoperative margin assessment. Here, we evaluate ex vivo breast tissue and corresponding quantified hematoxylin & eosin images to correlate optical scattering signatures to tissue composition stratified by patient characteristics. Adipose sites (213) were characterized by their cell area and density. All other benign and malignant sites (181) were quantified using a grid method to determine composition. The relationships between mean reduced scattering coefficient (〈μ's〉), and % adipose, % collagen, % glands, adipocyte cell area, and adipocyte density were investigated. These relationships were further stratified by age, menopausal status, body mass index (BMI), and breast density. We identified a positive correlation between 〈μ's〉 and % collagen and a negative correlation between 〈μ's〉 and age and BMI. Increased collagen corresponded to increased 〈μ's〉 variability. In postmenopausal women, 〈μ's〉 was similar regardless of fibroglandular content. Contributions from collagen and glands to 〈μ's〉 were independent and equivalent in benign sites; glands showed a stronger positive correlation than collagen to 〈μ's〉 in malignant sites. Our data suggest that scattering could differentiate highly scattering malignant from benign tissues in postmenopausal women. The relationship between scattering and tissue composition will support improved scattering models and technologies to enhance intraoperative optical margin assessment.