The growing importance of the “European Protocol for the Quality Control of the Physical and Technical Aspects of Mammography Screening, Part B: Digital Mammography” dictates the need to understand the prescribed threshold contrast sensitivity test. Observers following a 4-AFC paradigm, report the location of disks varying in contrast and diameter on multiple images of a CDMAM or similar phantom. Analysis provides a contrast threshold for each disk diameter. The goals of this study were to quantify the performance of new observers, compare it to published results, compare visual scoring with software scoring of the same images, and to quantify the major sources of variability. Digital phantom images, visual scorings by four expert readers, and CDCOM software were downloaded from the EUREF website. These images were read on a 3M Barco flat-panel monitor by 13 observers and scored by CDCOM. Scores were analyzed using the published method from the CDMAM-phantom 3.4 manual and a signal detection theory-based method. The average contrast sensitivities of the 13 study observers generally exceeded the published values by ~10%. The 95% confidence limits for the mean of 6 images from the published data vary from ±20.2% to ±41.8% of their respective means, the average being 31.2%. The average confidence limit for selected study observers is ±36%. Comparisons between software and human observer results using the prescribed method of analysis-revealed marked differences, particularly for small diameter targets. These differences are mitigated by signal-detection-theory analysis of both datasets. The large inter-observer variability and the substantial time required for human scoring supports the need to qualify a readily available software solution.