Regions of interest (ROIs) that are pointed to by overlaid markers (arrows, asterisks, etc.) in biomedical images
are expected to contain more important and relevant information than other regions for biomedical article
indexing and retrieval. We have developed several algorithms that localize and extract the ROIs by recognizing
markers on images. Cropped ROIs then need to be annotated with contents describing them best. In most cases
accurate textual descriptions of the ROIs can be found from figure captions, and these need to be combined
with image ROIs for annotation. The annotated ROIs can then be used to, for example, train classifiers that
separate ROIs into known categories (medical concepts), or to build visual ontologies, for indexing and retrieval
of biomedical articles.
We propose an algorithm that pairs visual and textual ROIs that are extracted from images and figure
captions, respectively. This algorithm based on dynamic time warping (DTW) clusters recognized pointers into
groups, each of which contains pointers with identical visual properties (shape, size, color, etc.). Then a rule-based
matching algorithm finds the best matching group for each textual ROI mention. Our method yields a
precision and recall of 96% and 79%, respectively, when ground truth textual ROI data is used.