Interferometric stability of polarization-entangled photons in quantum repeaters for long time intervals is an important capability for future scalable quantum networks linked over distances greater than hundreds of kilometers. A quantum memory node is a necessary component of the quantum repeater, where entanglement is prepared and swapped to extend entangled states from remote to distant nodes. Room temperature fluctuations can have significant effects on phase stability of the polarization states stored in the quantum memory. Although common-path stabilization in a quantum memory has been demonstrated, passive stabilization to room temperature variations has not been realized. Our approach to the quantum memory uses a single collective excitation encoded in two separate spatial modes in a cold ensemble of rubidium atoms. The two spatial modes are combined into a single path using the birefringence of two calcite crystals. However, normal lab temperature changes introduces a phase shift between the ordinary and extraordinary pathways on the order of 2π. We demonstrate passive temperature stabilization by alignment of the ordinary path in one crystal to the extraordinary path in the second crystal and vice versa. We show a phase stability on the order of ten hours by homodyne detection of classical light modes exiting the interferometer. We corroborate the phase stability of the quantum memory with a correlation measurement between polarization states of a signal photon generated at the formation of the collective atomic excitation and a retrieved idler photon during the destruction of the atomic excitation. We measure a Bell-CHSH parameter for both the unstable configuration and for the stable configuration. For an unstable calcite crystal configuration, we do not measure a violation of the Bell-CHSH inequality<sup>1</sup> (S≤2) with S = 1.42 ± 0.087. For the stable calcite crystal configuration, we measure a violation of Bell-CHSH inequality (S>2) with S = 2.48±0.099.