We present a high speed, phase-sensitive, line-scanning reflectance confocal interference microscope. We achieved rapid confocal imaging using a fast line-scan camera and quantitative phase imaging using off-axis digital holography on a 1D, line-by-line basis. In our prototype system, a He-Ne laser (~1.2 mW) was used to demonstrate the principle of operation. Using a 20 kHz line scan rate (1024 pixels per line scan), we achieved a video-rate frame rate of 20 Hz for 1024x500 pixel en-face confocal images (20 MHz total pixel rate). By using an objective lens of a NA 0.65, we achieved an axial and lateral resolution of ~3.5 micrometers and ~0.8 micrometers, respectively. By z-stack imaging of a custom silicon target with a stepped structure, we confirmed that the axial sectioning of the interference microscope is similar to that of a traditional line-scan confocal microscope (our microscope with the reference arm blocked). The utility of phase-sensitive holographic detection in line-scan confocal was demonstrated in two ways. First, using a custom axial height phantom fabricated using chrome deposition, we demonstrated variations in phase corresponding to heights in the 100 nm range with a contrast-to-noise ratio of ~31 dB. Second, we demonstrate digital refocusing of an out-of-focus holographic image. The mechanism of confocality in our line-scan system is 1D physical pinholing. Our ongoing work aims to add an additional mechanism of confocality by using low spatial coherence sources to impose interferometric pinholing.