Skeletal maturity is assessed visually by comparing hand radiographs to a standardized reference image atlas. Most
common are the methods by Greulich & Pyle and Tanner & Whitehouse. For computer-aided diagnosis (CAD), local
image regions of interest (ROI) such as the epiphysis or the carpal areas are extracted and evaluated. Heuristic
approaches trying to automatically extract, measure and classify bones and distances between bones suffer from the high
variability of biological material and the differences in bone development resulting from age, gender and ethnic origin.
Content-based image retrieval (CBIR) provides a robust solution without delineating and measuring bones. In this work,
epiphyseal ROIs (eROIS) of a hand radiograph are compared to previous cases with known age, mimicking a human
observer. Leaving-one-out experiments are conducted on 1,102 left hand radiographs and 15,428 metacarpal and
phalangeal eROIs from the publicly available USC hand atlas. The similarity of the eROIs is assessed by a combination
of cross-correlation, image distortion model, and Tamura texture features, yielding a mean error rate of 0.97 years and a
variance of below 0.63 years. Furthermore, we introduce a publicly available online-demonstration system, where
queries on the USC dataset as well as on uploaded radiographs are performed for instant CAD. In future, we plan to
evaluate physician with CBIR-CAD against physician without CBIR-CAD rather than physician vs. CBIR-CAD.
Radiological bone age assessment is based on local image regions of interest (ROI), such as the epiphysis or the area of
carpal bones. These are compared to a standardized reference and scores determining the skeletal maturity are calculated.
For computer-aided diagnosis, automatic ROI extraction and analysis is done so far mainly by heuristic approaches. Due
to high variations in the imaged biological material and differences in age, gender and ethnic origin, automatic analysis is
difficult and frequently requires manual interactions. On the contrary, epiphyseal regions (eROIs) can be compared to
previous cases with known age by content-based image retrieval (CBIR). This requires a sufficient number of cases with
reliable positioning of the eROI centers. In this first approach to bone age assessment by CBIR, we conduct leaving-oneout
experiments on 1,102 left hand radiographs and 15,428 metacarpal and phalangeal eROIs from the USC hand atlas.
The similarity of the eROIs is assessed by cross-correlation of 16x16 scaled eROIs. The effects of the number of eROIs,
two age computation methods as well as the number of considered CBIR references are analyzed. The best results yield
an error rate of 1.16 years and a standard deviation of 0.85 years. As the appearance of the hand varies naturally by up to
two years, these results clearly demonstrate the applicability of the CBIR approach for bone age estimation.
Increasing use of digital imaging processing leads to an enormous amount of imaging data. The access to picture
archiving and communication systems (PACS), however, is solely textually, leading to sparse retrieval results because
of ambiguous or missing image descriptions. Content-based image retrieval (CBIR) systems can improve the clinical
diagnostic outcome significantly. However, current CBIR systems are not able to integrate their results with clinical
workflow and PACS. Existing communication standards like DICOM and HL7 leave many options for implementation
and do not ensure full interoperability. We present a concept of the standardized integration of a CBIR system for the
radiology workflow in accordance with the Integrating the Healthcare Enterprise (IHE) framework. This is based on the
IHE integration profile 'Post-Processing Workflow' (PPW) defining responsibilities as well as standardized
communication and utilizing the DICOM Structured Report (DICOM SR). Because nowadays most of PACS and RIS
systems are not yet fully IHE compliant to PPW, we also suggest an intermediate approach with the concepts of the
CAD-PACS Toolkit. The integration is independent of the particular PACS and RIS system. Therefore, it supports the
widespread application of CBIR in radiological routine. As a result, the approach is exemplarily applied to the Image
Retrieval in Medical Applications (IRMA) framework.
Radiological bone age assessment is based on global or local image regions of interest (ROI), such as epiphyseal regions
or the area of carpal bones. Usually, these regions are compared to a standardized reference and a score determining the
skeletal maturity is calculated. For computer-assisted diagnosis, automatic ROI extraction is done so far by heuristic
approaches. In this work, we apply a high-level approach of scene analysis for knowledge-based ROI segmentation.
Based on a set of 100 reference images from the IRMA database, a so called structural prototype (SP) is trained. In this
graph-based structure, the 14 phalanges and 5 metacarpal bones are represented by nodes, with associated location,
shape, as well as texture parameters modeled by Gaussians. Accordingly, the Gaussians describing the relative
positions, relative orientation, and other relative parameters between two nodes are associated to the edges. Thereafter,
segmentation of a hand radiograph is done in several steps: (i) a multi-scale region merging scheme is applied to extract
visually prominent regions; (ii) a graph/sub-graph matching to the SP robustly identifies a subset of the 19 bones; (iii)
the SP is registered to the current image for complete scene-reconstruction (iv) the epiphyseal regions are extracted from
the reconstructed scene. The evaluation is based on 137 images of Caucasian males from the USC hand atlas. Overall,
an error rate of 32% is achieved, for the 6 middle distal and medial/distal epiphyses, 23% of all extractions need
adjustments. On average 9.58 of the 14 epiphyseal regions were extracted successfully per image. This is promising for
further use in content-based image retrieval (CBIR) and CBIR-based automatic bone age assessment.
In this work, a concept for coupling a system for content-based image retrieval in medical applications (IRMA) with
hospital information systems is presented. We aim at improving the work flow of radiologists and evaluating the
recognition performance of the IRMA system in clinical routine. The integration is designed such that a failure of IRMA
does not affect the routine operation of the other systems. The coupling is realized by generic communication modules
with the radiology information system, and the picture archiving and communication system (PACS) over the standard
protocols Digital Imaging and Communications in Medicine (DICOM) and Health Layer 7 (HL7). An optional plug-in
for the radiological viewing station further enhances the usability. Based on this concept, the pre-fetching of relevant
images for recurrent examinations is improved. When an examination is scheduled, all previous images of the patient are
read by the IRMA system with DICOM query/retrieve. If the images were not present before in our database, features
are extracted, stored, and indexed. After the acquisition of new images from the imaging modality, the new images are
automatically retrieved by the IRMA system with DICOM query/retrieve and similar images are selected based on the
stored global signatures. These images are then loaded into the online storage of the PACS and are available for
diagnostic purposes together with those images already pre-selected by the PACS. Thus the radiologist can avoid further
delays resulting from manually fetching further images from archives which have not been automatically selected by
alphanumerical meta data. In addition, he is able to sort all fetched images by the computed IRMA-similarity.
Furthermore, the hanging of images in the viewing software is planned to be organized by IRMA suggestions
automatically, further shortening the time for the examination and reducing manual interactions. Based on the generality
of our integration concept, a CBIR-based second opinion to support the diagnostics, and computer-based training of
radiologists will be established in near future.
The content of medical images can often be described as a composition of relevant objects with distinct relationships. Each single object can then be represented as a graph node, and local features of the objects are associated as node attributes, e.g. the centroid coordinates. The relations between these objects are represented as graph edges with annotated relational features, e.g. their relative size. Nodes and edges build an attributed relational graph (ARG). For a given setting, e.g. a hand radiograph, a generalization of the relevant objects, e.g. individual bone segments, can be obtained by the statistical distributions of all attributes computed from training images. These yield a structural prototype graph consisting of one attributed node per relevant object and of their relations represented as attributed edges. In contrast to the ARG, the mean and standard deviation of each local or relational feature are used to annotate the prototype nodes or edges, respectively. The prototype graph can then be used to identify the generalized objects in new images. As new image content is represented by hierarchical attributed region adjacency graphs (HARAGs) which are obtained by region-growing, the task of object or scene identification corresponds to the problem of inexact sub-graph matching between a small prototype and the current HARAG. For this purpose, five approaches are evaluated in an example application of bone-identification in 96 radiographs: Nested Earth Mover's Distance, Graph Edit Distance, a Hopfield Neural Network, Pott's Mean Field Annealing and Similarity Flooding. The discriminative power of 34 local and 12 relational features is judged for each object by sequential forward selection. The structural prototypes improve recall by up to 17% in comparison to the approach without relational information.
Proc. SPIE. 5748, Medical Imaging 2005: PACS and Imaging Informatics
KEYWORDS: Human-machine interfaces, Visualization, Databases, Image segmentation, Radiography, Image analysis, Bone, Medical imaging, Information visualization, Picture Archiving and Communication System
The classification and measuring of objects in medical images is important in radiological diagnostics and education, especially when using large databases as knowledge resources, for instance a picture archiving and communication system (PACS). The main challenge is the modeling of medical knowledge and the diagnostic context to label the sought objects. This task is referred to as closing the semantic gap between low-level pixel information and high level application knowledge. This work describes an approach which allows labeling of a-priori unknown objects in an intuitive way.
Our approach consists of four main components. At first an image is completely decomposed into all visually relevant partitions on different scales. This provides a hierarchical organized set of regions. Afterwards, for each of the obtained regions a set of descriptive features is computed. In this data structure objects are represented by regions with characteristic attributes. The actual object identification is the formulation of a query. It consists of attributes on which intervals are defined describing those regions that correspond to the sought objects. Since the objects are a-priori unknown, they are described by a medical expert by means of an intuitive graphical user interface (GUI). This GUI is the fourth component.
It enables complex object definitions by browsing the data structure and examinating the attributes to formulate the query. The query is executed and if the sought objects have not been identified its parameterization is refined.
By using this heuristic approach, object models for hand radiographs have been developed to extract bones from a single hand in different anatomical contexts. This demonstrates the applicability of the labeling concept. By using a rule for metacarpal bones on a series of 105 images, this type of bone could be retrieved with a precision of 0.53 % and a recall of 0.6%.
Content-based image retrieval requires a formal description of visual information. In medical applications, all relevant biological objects have to be represented by this description. Although color as the primary feature has proven successful in publicly available retrieval systems of general purpose, this description is not applicable to most medical images. Additionally, it has been shown that global features characterizing the whole image do not lead to acceptable results in the medical context or that they are only suitable for specific applications. For a general purpose content-based comparison of medical images, local, i.e. regional features that are collected on multiple scales must be used. A hierarchical attributed region adjacency graph (HARAG) provides such a representation and transfers image comparison to graph matching. However, building a HARAG from an image requires a restriction in size to be computationally feasible while at the same time all visually plausible information must be preserved. For this purpose, mechanisms for the reduction of the graph size are presented. Even with a reduced graph, the problem of graph matching remains NP-complete. In this paper, the Similarity Flooding approach and Hopfield-style neural networks are adapted from the graph matching community to the needs of HARAG comparison. Based on synthetic image material build from simple geometric objects, all visually similar regions were matched accordingly showing the framework's general applicability to content-based image retrieval of medical images.
Proc. SPIE. 5150, Visual Communications and Image Processing 2003
KEYWORDS: Content based image retrieval, Databases, Image processing, Computing systems, Feature extraction, Medical imaging, Image retrieval, Image storage, Algorithm development, Picture Archiving and Communication System
We describe a platform for the implementation of a system for content-based image retrieval in medical applications (IRMA). To cope with the constantly evolving medical knowledge, the platform offers a flexible feature model to store and uniformly access all feature types required within a multi-step retrieval approach. A structured generation history for each feature allows the automatic identification and re-use of already computed features. The platform uses directed acyclic graphs composed of processing steps and
control elements to model arbitrary retrieval algorithms. This visually intuitive, data-flow oriented representation vastly improves the interdisciplinary communication between computer scientists and physicians during the development of new retrieval algorithms.
The execution of the graphs is fully automated within the platform.
Each processing step is modeled as a feature transformation. Due to a high degree of system transparency, both the implementation and the
evaluation of retrieval algorithms are accelerated significantly.
The platform uses a client-server architecture consisting of a central database, a central job scheduler, instances of a daemon
service, and clients which embed user-implemented feature ansformations. Automatically distributed batch processing and distributed feature storage enable the cost-efficient use of an existing workstation cluster.
Proc. SPIE. 5033, Medical Imaging 2003: PACS and Integrated Medical Information Systems: Design and Evaluation
KEYWORDS: Content based image retrieval, Human-machine interfaces, Transparency, Imaging systems, Databases, Image segmentation, Image processing, Feature extraction, Medical imaging, Picture Archiving and Communication System
Picture archiving and communication systems (PACS) aim to efficiently provide the radiologists with all images in a suitable quality for diagnosis. Modern standards for digital imaging and communication in medicine (DICOM) comprise alphanumerical descriptions of study, patient, and technical parameters. Currently, this is the only information used to select relevant images within PACS. Since textual descriptions insufficiently describe the great variety of details in medical images, content-based image retrieval (CBIR) is expected to have a strong impact when integrated into PACS. However, existing CBIR approaches usually are limited to a distinct modality, organ, or diagnostic study. In this state-of-the-art report, we present first results implementing a general approach to content-based image retrieval in medical applications (IRMA) and discuss its integration into PACS environments. Usually, a PACS consists of a DICOM image server and several DICOM-compliant workstations, which are used by radiologists for reading the images and reporting the findings. Basic IRMA components are the relational database, the scheduler, and the web server, which all may be installed on the DICOM image server, and the IRMA daemons running on distributed machines, e.g., the radiologists’ workstations. These workstations can also host the web-based front-ends of IRMA applications. Integrating CBIR and PACS, a special focus is put on (a) location and access transparency for data, methods, and experiments, (b) replication transparency for methods in development, (c) concurrency transparency for job processing and feature extraction, (d) system transparency at method implementation time, and (e) job distribution transparency when issuing a query. Transparent integration will have a certain impact on diagnostic quality supporting both evidence-based medicine and case-based reasoning.
In this paper we describe the construction of hierarchical feature clustering and show how to overcome general problems of region growing algorithms such as seed point selection and processing order. Access to medical knowledge inherent in medical image databases requires content-based descriptions to allow non-textual retrieval, e.g., for comparison, statistical inquiries, or education. Due to varying medical context and questions, data structures for image description must provide all visually perceivable regions and their topological relationships, which poses one of the major problems for content extraction. In medical applications main criteria for segmenting images are local features such as texture, shape, intensity extrema, or gray values. For this new approach, these features are computed pixel-based and neighboring pixels are merged if the Euclidean distance of corresponding feature vectors is below a threshold. Thus, the planar adjacency of clusters representing connected image partitions is preserved. A cluster hierarchy is obtained by iterating and recording the adjacency merging. The resulting inclusion and neighborhood relations of the regions form a hierarchical region adjacency graph. This graph represents a multiscale image decomposition and therefore an extensive content description. It is examined with respect to application in daily routine by testing invariance against transformation, run time behavior, and visual quality For retrieval purposes, a graph can be matched with graphs of other images, where the quality of the matching describes the similarity of the images.