Emerging underwater optical imaging and sensing applications rely on phase-sensitive detection to provide added functionality and improved sensitivity. However, underwater turbulence introduces spatio-temporal variations in the refractive index of water which can degrade the performance of these systems. Although the influence of turbulence on traditional, non-interferometric imaging has been investigated, its influence on the optical phase remains poorly understood. Nonetheless, a thorough understanding of the spatio-temporal dynamics of the optical phase of light passing through underwater turbulence are crucial to the design of phase-sensitive imaging and sensing systems. To address this concern, we combined underwater imaging with high speed holography to provide a calibrated characterization of the effects of turbulence on the optical phase. By measuring the modulation transfer function of an underwater imaging system, we were able to calibrate varying levels of optical turbulence intensity using the Simple Underwater Imaging Model (SUIM). We then used high speed holography to measure the temporal dynamics of the optical phase of light passing through varying levels of turbulence. Using this method, we measured the variance in the amplitude and phase of the beam, the temporal correlation of the optical phase, and recorded the turbulence induced phase noise as a function of frequency. By bench marking the effects of varying levels of turbulence on the optical phase, this work provides a basis to evaluate the real-world potential of emerging underwater interferometric sensing modalities.