We present an interpretable end-to-end computer-aided detection and diagnosis tool for pulmonary nodules on computed tomography (CT) using deep learning-based methods. The proposed network consists of a nodule detector and a nodule malignancy classifier. We used RetinaNet to train a nodule detector using 7,607 slices containing 4,234 nodule annotations and validated it using 2,323 slices containing 1,454 nodule annotations drawn from the LIDC-IDRI dataset. The average precision for the nodule class in the validation set reached 0.24 at an intersection over union (IoU) of 0.5. The trained nodule detector was externally validated using a UCLA dataset. We then used a hierarchical semantic convolutional neural network (HSCNN) to classify whether a nodule was benign or malignant and generate semantic (radiologist-interpretable) features (e.g., mean diameter, consistency, margin), training the model on 149 cases with diagnostic CTs collected from the same UCLA dataset. A total of 149 nodule-centered patches from the UCLA dataset were used to train the HSCNN. Using 5-fold cross validation and data augmentation, the mean AUC and mean accuracy in the validation set for predicting nodule malignancy achieved 0.89 and 0.74, respectively. Meanwhile, the mean accuracy for predicting nodule mean diameter, consistency, and margin were 0.59, 0.74, and 0.75, respectively. We have developed an initial end-to-end pipeline that automatically detects nodules ≥ 5 mm on CT studies and labels identified nodules with radiologist-interpreted features automatically.
SC1295: From Analytic to Clinical Validation: Moving AI/ML into Practice
Artificial Intelligence (AI) is increasingly being used in a wide variety of medical imaging applications. Most of the focus, however, is on algorithm and scheme development, but this is only part of the picture. In order to have an impact on clinical decision making, workflow and patient care these AI tools must be evaluated using real-world cases and actual clinical providers that are expected to use them in routine care. The techniques used to conduct these types of studies are less well known in this field thus investigators need to be trained the proper study design and analysis methods.
This course will cover basic principles, techniques, and process for validating models developed using artificial intelligence (AI)/machine learning (ML) techniques. The primary goal of this course is to help the audience understand and apply fundamental principles related to designing, executing, and interpreting model evaluation studies. The course will be organized around two parts: analytic validation and clinical validation. In the first half, the audience will be exposed to approaches for performing a technical validation of a prediction model, including different study designs, appropriate statistical tests, metrics, dataset considerations, and decision curve analysis. The second half will cover the process of undertaking clinical validation that would address real-world use of models, regulatory and deployment issues. Topics include workflow integration, prospective clinical trials, reader impact studies, and regulatory approvals. Examples will focus on imaging-related models that are drawn from literature and the instructors’ personal experiences in prognostic modeling, computer-aided diagnosis, and imaging biomarker development.