Neuroanatomical pathways form the basis for functional activity of brain circuits. In the past, we developed a polarization-sensitive optical coherence tomography with serial scanning to achieve large-scale brain imaging. The system was able to visualize 3D fiber tracts of ~20 um in diameter. To investigate the neuroanatomical pathways at finer scales, we have now built a polarization-maintaining fiber based ultra-high resolution polarization-sensitive optical coherence microscope (PS-OCM) at 1300 nm. The PS-OCM has an axial resolution of 3.5 um in tissue. The detection setup consists of two spectrometers, acquiring spectral interference on orthogonal polarization channels. With a single measurement, the setup generates four contrasts: reflectivity, cross-polarization, retardance and optic axis orientation. To investigate the capability of PS-OCM at different resolutions, we used three microscope objectives that yield lateral resolutions of 6.0 um, 3.4 um and 1.3 um. Blocks of formalin fixed mouse brain and human brain were scanned. The cross-polarization and retardance images clearly depict the neuronal fiber structures, which are comparable with that generated by the maximum projection of volumetric reflectivity data. The optic axis orientation quantifies the in-plane fiber orientation. With the lateral resolution of 1.3 um, the retardance contrast is weak in white matter due to the shallow depth of focus. Overall, the ultra-high resolution PS-OCM provides a new tool to reveal neuroanatomical maps in the brain at cellular resolution.