Quantitative polarized light microscopy (qPLM) is a popular tool for the investigation of birefringent architectures in biological tissues. Collagen, the most abundant protein in mammals, is such a birefringent material. Interpretation of results of qPLM in terms of collagen network architecture and anisotropy is challenging, because different collagen networks may yield equal qPLM results. We created a model and used the linear optical behavior of collagen to construct a Jones or Mueller matrix for a histological cartilage section in an optical qPLM train. Histological sections of tendon were used to validate the basic assumption of the model. Results show that information on collagen densities is needed for the interpretation of qPLM results in terms of collagen anisotropy. A parameter that is independent of the optical system and that measures collagen fiber anisotropy is introduced, and its physical interpretation is discussed. With our results, we can quantify which part of different qPLM results is due to differences in collagen densities and which part is due to changes in the collagen network. Because collagen fiber orientation and anisotropy are important for tissue function, these results can improve the biological and medical relevance of qPLM results.