Enriching text with images and colored light

Dragan Sekulovski; Gijs Geleijnse; Bram Kater; Jan Korst; Steffen Pauws; Ramon Clout

doi:10.1117/12.766421

28 January 2008 Enriching text with images and colored light

Dragan Sekulovski, Gijs Geleijnse, Bram Kater, Jan Korst, Steffen Pauws, Ramon Clout

Proceedings Volume 6820, Multimedia Content Access: Algorithms and Systems II; 682004 (2008) https://doi.org/10.1117/12.766421
Event: Electronic Imaging, 2008, San Jose, California, United States

Abstract

We present an unsupervised method to enrich textual applications with relevant images and colors. The images are collected by querying large image repositories and subsequently the colors are computed using image processing. A prototype system based on this method is presented where the method is applied to song lyrics. In combination with a lyrics synchronization algorithm the system produces a rich multimedia experience. In order to identify terms within the text that may be associated with images and colors, we select noun phrases using a part of speech tagger. Large image repositories are queried with these terms. Per term representative colors are extracted using the collected images. Hereto, we either use a histogram-based or a mean shift-based algorithm. The representative color extraction uses the non-uniform distribution of the colors found in the large repositories. The images that are ranked best by the search engine are displayed on a screen, while the extracted representative colors are rendered on controllable lighting devices in the living room. We evaluate our method by comparing the computed colors to standard color representations of a set of English color terms. A second evaluation focuses on the distance in color between a queried term in English and its translation in a foreign language. Based on results from three sets of terms, a measure of suitability of a term for color extraction based on KL Divergence is proposed. Finally, we compare the performance of the algorithm using either the automatically indexed repository of Google Images and the manually annotated Flickr.com. Based on the results of these experiments, we conclude that using the presented method we can compute the relevant color for a term using a large image repository and image processing.

Citation Download Citation

Dragan Sekulovski, Gijs Geleijnse, Bram Kater, Jan Korst, Steffen Pauws, and Ramon Clout "Enriching text with images and colored light", Proc. SPIE 6820, Multimedia Content Access: Algorithms and Systems II, 682004 (28 January 2008); https://doi.org/10.1117/12.766421

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available