Facile analysis of the architecture of the mammalian brain is key to understanding how brain function emerges during development and dysregulated in disorders including neurodegeneration. Immunolabeling of mammalian brain tissue, especially scarce human brain tissue, is time-consuming, can introduce sample-to-sample variation, and is not compatible with live imaging. We report joint optimization of polarization-resolved label-free imaging and deep learning to map brain architecture. We visualize diverse structures in human brain tissue by mapping optical properties of density, birefringence, orientation, and scattering. We design computationally efficient variants of U-Nets to predict tract distribution and cell types from intrinsic optical properties of the tissue.