Large-scale medical image annotation with crowd-powered algorithms

Eric Heim; Tobias Roß; Alexander Seitel; Keno März; Bram Stieltjes; Matthias Eisenmann; Johannes Lebert; Jasmin Metzger; Gregor Sommer; Alexander W. Sauter; Fides Regina Schwartz; Andreas Termer; Felix Wagner; Hannes Götz Kenngott; Lena Maier-Hein

doi:10.1117/1.JMI.5.3.034002

8 September 2018 Large-scale medical image annotation with crowd-powered algorithms

Eric Heim, Tobias Roß, Alexander Seitel, Keno März, Bram Stieltjes, Matthias Eisenmann, Johannes Lebert, Jasmin Metzger, Gregor Sommer, Alexander W. Sauter, Fides Regina Schwartz, Andreas Termer, Felix Wagner, Hannes Götz Kenngott, Lena Maier-Hein

Author Affiliations +

Journal of Medical Imaging, Vol. 5, Issue 3, 034002 (September 2018). https://doi.org/10.1117/1.JMI.5.3.034002

Abstract

Accurate segmentations in medical images are the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotated reference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3-D segmentation algorithm integrated into a medical imaging platform. In a pilot study of liver segmentation using a publicly available dataset of computed tomography scans, we show that the crowd is able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowds need significantly more time for the annotation of a slice, the annotation rate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.

Citation Download Citation

Eric Heim, Tobias Roß, Alexander Seitel, Keno März, Bram Stieltjes, Matthias Eisenmann, Johannes Lebert, Jasmin Metzger, Gregor Sommer, Alexander W. Sauter, Fides Regina Schwartz, Andreas Termer, Felix Wagner, Hannes Götz Kenngott, and Lena Maier-Hein "Large-scale medical image annotation with crowd-powered algorithms," Journal of Medical Imaging 5(3), 034002 (8 September 2018). https://doi.org/10.1117/1.JMI.5.3.034002

Received: 13 April 2018; Accepted: 26 July 2018; Published: 8 September 2018

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available