An exploration of uncertainty information for segmentation quality assessment

Katharina Hoebel; Vincent Andrearczyk; Andrew Beers; Jay Patel; Ken Chang; Adrien Depeursinge; Henning Müller; Jayashree Kalpathy-Cramer

doi:10.1117/12.2548722

Translator Disclaimer

10 March 2020 An exploration of uncertainty information for segmentation quality assessment

Katharina Hoebel, Vincent Andrearczyk, Andrew Beers, Jay Patel, Ken Chang, Adrien Depeursinge, Henning Müller, Jayashree Kalpathy-Cramer

Author Affiliations +

Proceedings Volume 11313, Medical Imaging 2020: Image Processing; 113131K (2020) https://doi.org/10.1117/12.2548722
Event: SPIE Medical Imaging, 2020, Houston, Texas, United States

Abstract

Including uncertainty information in the assessment of a segmentation of pathologic structures on medical images, offers the potential to increase trust into deep learning algorithms for the analysis of medical imaging. Here, we examine options to extract uncertainty information from deep learning segmentation models and the influence of the choice of cost functions on these uncertainty measures. To this end we train conventional UNets without dropout, deep UNet ensembles, and Monte-Carlo (MC) dropout UNets to segment lung nodules on low dose CT using either soft Dice or weighted categorical cross-entropy (wcc) as loss functions. We extract voxel-wise uncertainty information from UNet models based on softmax maximum probability and from deep ensembles and MC dropout UNets using mean voxel-wise entropy. Upon visual assessment, areas of high uncertainty are localized in the periphery of segmentations and are in good agreement with incorrectly labelled voxels. Furthermore, we evaluate how well uncertainty measures correlate with segmentation quality (Dice score). Mean uncertainty over the segmented region (Ulabelled) derived from conventional UNet models does not show a strong quantitative relationship with the Dice score (Spearman correlation coefficient of -0.45 for the soft Dice vs -0.64 for the wcc model respectively). By comparison, image-level uncertainty measures derived from soft Dice as well as wcc MC UNet and deep UNet ensemble models correlate well with the Dice score. In conclusion, using uncertainty information offers ways to assess segmentation quality fully automatically without access to ground truth. Models trained using weighted categorical cross-entropy offer more meaningful uncertainty information on a voxel-level.

Conference Presentation

Citation Download Citation

Katharina Hoebel, Vincent Andrearczyk, Andrew Beers, Jay Patel, Ken Chang, Adrien Depeursinge, Henning Müller, and Jayashree Kalpathy-Cramer "An exploration of uncertainty information for segmentation quality assessment", Proc. SPIE 11313, Medical Imaging 2020: Image Processing, 113131K (10 March 2020); https://doi.org/10.1117/12.2548722

ACCESS THE FULL ARTICLE