Polarized light is commonly used to detect optical anisotropies, such as birefringence, in tissues. This optical anisotropy is often attributed to underlying structural anisotropy in tissue, which may originate from regularly aligned collagen fibers. In these cases, the optical anisotropy, such as birefringence, is interpreted as a relative measure of the structural anisotropy of the collagen fibers. However, the relative amplitude of optical anisotropy depends on factors other than fiber orientation, and few models allow quantitative interpretation of absolute measures of true fiber orientation distribution from the optical signal. Our model uses the Mie solution to scattering of linearly polarized light from infinite cylindrical scatterers. The model is expanded to include populations of scatterers with physiologically relevant size and orientation distributions. We investigated the influences of fiber diameter, orientation distribution, and wavelength on the back-scattering signal with our computational model, and used these results to extract structural information from experimental fiber phantoms and bovine tendon. Our results demonstrated that by fitting our model to the experimental data using limited assumptions, we could extract fiber orientation distributions and diameters that were comparable to those found in scanning electron microscope images of the same fiber sample. We found a higher alignment of fibers in the bovine tendon sample, and the extracted fiber diameter was within the expected physiological range. Our model showed that the amplitude of optical anisotropy can vary widely due to factors other than the orientation distribution of fiber structures, including index of refraction, and therefore should not be taken as a sole indicator of structural anisotropy. This work highlights that the accuracy of model assumptions plays a crucial role in extracting quantitative structural information from optical anisotropy.