We examine some fundamental theoretical limits on the ability of practical digital holography (DH) systems to resolve detail in an image. Unlike conventional diffraction-limited imaging systems, where a projected image of the limiting aperture is used to define the system performance, there are at least three major effects that determine the performance of a DH system: (i) The spacing between adjacent pixels on the CCD, (ii) an averaging effect introduced by the finite size of these pixels, and (iii) the finite extent of the camera face itself. Using a theoretical model, we define a single expression that accounts for all these physical effects. With this model, we explore several different DH recording techniques: off-axis and inline, considering both the dc terms, as well as the real and twin images that are features of the holographic recording process. Our analysis shows that the imaging operation is shift variant and we demonstrate this using a simple example. We examine how our theoretical model can be used to optimize CCD design for lensless DH capture. We present a series of experimental results to confirm the validity of our theoretical model, demonstrating recovery of super-Nyquist frequencies for the first time.