Presentation of images similar to a new unknown lesion as a reference can be helpful in medical image diagnosis and treatment planning. We have been investigating a method to determine similarity of breast masses as an image retrieval index for an intelligent image analytic system that may support radiologists’ efficient image interpretation. In order to retrieve perceptually similar images, we have obtained subjective similarity ratings from expert radiologists, which were then used in similarity space modeling and training deep neural networks. In this study, we investigated the use of convolutional neural network to model the similarity space for retrieval of diagnostically relevant reference images and also to directly estimate similarity ratings for pairs of images. The preliminary results show that retrieval performance was slightly better in similarity space modeling method than direct estimation method. These results indicate the potential usefulness of the proposed methods for retrieval of reference images as diagnostic assistance.
Presentation of reference images that are similar to a query image can be helpful in medical image diagnosis and treatment planning. The purpose of this study is to investigate a method for retrieving relevant images of breast masses on mammograms as a diagnostic reference. In our previous studies, subjective similarities for pairs of masses were obtained from experienced radiologists and used as the gold standard for retrieving visually similar images. By use of multidimensional scaling, a subjective similarity space was spanned so that masses that were placed close to a query image can be retrieved as reference images. This method, however, required manual outlines of masses for image feature determination. In this study, we modelled this similarity space using convolutional neural network. The result was evaluated using the leave-one-out cross validation method in terms of the correlation between the subjective ratings and determined similarity measures. The relevance of retrieved images was also evaluated in terms of the precision, which is the fraction of pathology matched images in the retrieved images. The correlation coefficient between the subjective ratings and the determined similarity measure was moderate with 0.735, which was slightly lower than those of the previous methods. The average precision was high with 0.852 when the most similar image was retrieved, which was higher than those in the previous studies. The results indicate the potential usefulness of the proposed method in similar image retrieval of breast masses on mammograms.
Periodic breast cancer screening with mammography is considered effective in decreasing breast cancer mortality. For screening programs to be successful, an intelligent image analytic system may support radiologists’ efficient image interpretation. In our previous studies, we have investigated image retrieval schemes for diagnostic references of breast lesions on mammograms and ultrasound images. Using a machine learning method, reliable similarity measures that agree with radiologists’ similarity were determined and relevant images could be retrieved. However, our previous method includes a feature extraction step, in which hand crafted features were determined based on manual outlines of the masses. Obtaining the manual outlines of masses is not practical in clinical practice and such data would be operator-dependent. In this study, we investigated a similarity estimation scheme using a convolutional neural network (CNN) to skip such procedure and to determine data-driven similarity scores. By using CNN as feature extractor, in which extracted features were employed in determination of similarity measures with a conventional 3-layered neural network, the determined similarity measures were correlated well with the subjective ratings and the precision of retrieving diagnostically relevant images was comparable with that of the conventional method using handcrafted features. By using CNN for determination of similarity measure directly, the result was also comparable. By optimizing the network parameters, results may be further improved. The proposed method has a potential usefulness in determination of similarity measure without precise lesion outlines for retrieval of similar mass images on mammograms.