Radiologists and cardiologists today have to view large amounts of imaging data relatively quickly leading to eye fatigue. Further, they have only limited access to clinical information relying mostly on their visual interpretation of imaging studies for their diagnostic decisions. In this paper, we present Medical Sieve, an automated cognitive assistant for radiologists and cardiologists designed to help in their clinical decision-making. The sieve is a clinical informatics system that collects clinical, textual and imaging data of patients from electronic health records systems. It then analyzes multimodal content to detect anomalies if any, and summarizes the patient record collecting all relevant information pertinent to a chief complaint. The results of anomaly detection are then fed into a reasoning engine which uses evidence from both patient-independent clinical knowledge and large-scale patient-driven similar patient statistics to arrive at potential differential diagnosis to help in clinical decision making. In compactly summarizing all relevant information to the clinician per chief complaint, the system still retains links to the raw data for detailed review providing holistic summaries of patient conditions. Results of clinical studies in the domains of cardiology and breast radiology have already shown the promise of the system in differential diagnosis and imaging studies summarization.
In this paper we propose a new method for abnormality detection in medical images which is based on the notion of medical saliency. The proposed method is general and is suitable for a variety of tasks related to detection of: 1) lesions and microcalcifications (MCC) in mammographic images, 2) stenoses in angiographic images, 3) lesions found in magnetic resonance (MRI) images of brain. The main idea of our approach is that abnormalities manifest as rare events, that is, as salient areas compared to normal tissues. We define the notion of medical saliency by combining local patch information from the lightness channel with geometric shape local descriptors. We demonstrate the efficacy of the proposed method by applying it to various modalities, and to various abnormality detection problems. Promising results are demonstrated for detection of MCC and of masses in mammographic images, detection of stenoses in angiography images, and detection of lesions in brain MRI. We also demonstrate how the proposed automatic abnormality detection method can be combined with a system that performs supervised classification of mammogram images into benign or malignant/premalignant MCC's. We use a well known DDSM mammogram database for the experiment on MCC classification, and obtain 80% accuracy in classifying images containing premalignant MCC versus benign ones. In contrast to supervised detection methods, the proposed approach does not rely on ground truth markings, and, as such, is very attractive and applicable for big corpus image data processing.
To overcome operator dependency and to increase diagnosis accuracy in breast ultrasound (US), a lot of effort has been devoted to developing computer-aided diagnosis (CAD) systems for breast cancer detection and classification. Unfortunately, the efficacy of such CAD systems is limited since they rely on correct automatic lesions detection and localization, and on robustness of features computed based on the detected areas. In this paper we propose a new approach to boost the performance of a Machine Learning based CAD system, by combining visual and clinical data from patient files. We compute a set of visual features from breast ultrasound images, and construct the textual descriptor of patients by extracting relevant keywords from patients' clinical data files. We then use the Multiple Kernel Learning (MKL) framework to train SVM based classifier to discriminate between benign and malignant cases. We investigate different types of data fusion methods, namely, early, late, and intermediate (MKL-based) fusion. Our database consists of 408 patient cases, each containing US images, textual description of complaints and symptoms filled by physicians, and confirmed diagnoses. We show experimentally that the proposed MKL-based approach is superior to other classification methods. Even though the clinical data is very sparse and noisy, its MKL-based fusion with visual features yields significant improvement of the classification accuracy, as compared to the image features only based classifier.