Content-based video retrieval from archived image/video is a very attractive capability of modern intelligent video
surveillance systems. This paper presents an innovative Semantic-Based Video Indexing and Retrieval (SBVIR) software
toolkit to help users of intelligent video surveillance to easily and rapidly search the content of large video archives to
conduct video-based forensic and image intelligence. Tailored for maritime environment, SBVIR is suited for
surveillance applications in harbor, sea shores, or around ships. The system comprises two major modules: a video
analytic module that performs automatic target detection, tracking, classification, activities recognition, and a retrieval
module that performs data indexing, and information retrieval. SBVIR is capable of detecting and tracking objects from
multiple cameras robustly in condition of dynamic water background and illumination changes. The system provides
hierarchical target classification among a large ontology of watercraft classes, and is capable of recognizing a variety of
boat activities. Video retrieval is achieved with both query-by-keyword and query-by-example. Users can query video
content using semantic concepts selected from a large dictionary of objects and activities, display the history linked to a
given target/activity, and search for anomalies. The user can interact with the system and provide feedbacks to tune the
system for improved accuracy and relevance of retrieved data.
SBVIR has been tested for real maritime surveillance scenarios and shown to be able to generate highly-semantic
metadata tags that can be used during the retrieval to provide user with relevant and accurate data in real-time.