The widely used multi-reader multi-case ROC study design for comparing imaging modalities is the fullycrossed (FC) design: every reader reads every case of both modalities. In this work, we investigate the paired split-plot (PSP) designs that allow for reduced cost and increased flexibility compared to the FC design. In the PSP design, patient images from two modalities are read by the same readers, thereby the readings are paired across modalities. However, within each modality, not every reader reads every case. Instead, both the readers and the cases are partitioned into a number of groups and each group of readers read their own group of cases - a split-plot design. Using the U-statistic based variance analysis for AUC (i.e., area under the ROC curve), we show analytically that, with a fixed number of readings per reader, substantial statistical efficiency can be gained by the PSP design as compared to the FC design. Equivalently, we show that the PSP design can achieve the same statistical power as the FC design with substantially reduced number of readings. However, the efficiency/power gain of the PSP design comes with the increased cost of collecting a larger number of truthverified patient cases than the FC design. This means that one can trade off between different sources of cost and choose a least burdensome design. We demonstrate the advantages of the PSP design with a real-world reader study for the comparison of full field digital mammography with screen-film mammography.
Weijie Chen, Qi Gong, and Brandon D. Gallas, "Efficiency gain of paired split-plot designs in MRMC ROC studies," Proc. SPIE 10577, Medical Imaging 2018: Image Perception, Observer Performance, and Technology Assessment, 105770F (Presented at SPIE Medical Imaging: February 11, 2018; Published: 7 March 2018); https://doi.org/10.1117/12.2293741.
Conference Presentations are recordings of oral presentations given at SPIE conferences and published as part of the conference proceedings. They include the speaker's narration along with a video recording of the presentation slides and animations. Many conference presentations also include full-text papers. Search and browse our growing collection of more than 14,000 conference presentations, including many plenary and keynote presentations.