Mammary epithelial cell (MEC) organoids in 3D culture recapitulate features of breast ducts in vivo. OCT has the ability to monitor the evolution of MEC organoids non-invasively and longitudinally. The anti-cancer drug Doxorubicin (Dox) is able to inhibit proliferation of cancer cells and has been widely used for chemotherapy of breast cancers; while environmental toxins implicated in breast cancer such as estrogen regulates mammary tumor growth and stimulates the proliferation and metastatic potential of breast cancers. Here we propose a quantitative method for measuring motility of breast cells in 3D cultures based upon OCT speckle fluctuation spectroscopy. The metrics of the inverse power-law exponent (α) and fractional modulation amplitude (M) were extracted from speckle fluctuation spectra. These were used to quantify the responses of MEC organoids to Dox, and estrogen. We investigated MEC organoids comprised of two different MEC lines: MCF10DCIS.com exposed to Dox, and MCF7 exposed to estrogen. We found an increase (p<0.001) in α of MEC along time (t=0, 1 hour, 24 hours, 48 hours and 6 days) at each dose of Dox (0, 1 μM and 10 μM), indicating lower fluctuation intensity at higher frequencies. We also observed a decrease (p<0.001) in M for increasing time. However, both α and M of MCF7 treated with estrogen (0, 1 nM and 10 nM) exhibited the opposite trend along time. This novel technology provides rapid and non-invasive measurements of the effects of toxicants on MEC motility for understanding breast cancer development and assessing anti-cancer drugs.