This paper introduces the concept of a contrast budget for a head-mounted display, i.e., a list of contrast values of separate contrast-degrading phenomena in the system. The basic phenomena are dynamic range, diffraction, scattered light, and ghost images. By calculating a contrast budget it becomes clear whether or not the display-system contrast will meet specified criteria and which factors contribute the most to the degradation of contrast in the system. An example of a contrast budget is given for the NomadTM head-mounted display system. In this system dynamic range, scanner scattering, and beamsplitter stray light are the dominant contributors, but the overall contrast is comfortably above specification.