23 June 2003 Multimetric evaluation protocol for user-assisted video object extraction systems
Author Affiliations +
Proceedings Volume 5150, Visual Communications and Image Processing 2003; (2003) https://doi.org/10.1117/12.509857
Event: Visual Communications and Image Processing 2003, 2003, Lugano, Switzerland
In this paper, a multi-metric evaluation protocol is proposed to evaluate performance of user-assisted video object extraction systems. Evaluation metrics are the essential element in performance evaluation methodology. Recent works on video object segmentation/extraction are mostly restricted to a single objective metric to judge the overall performance of algorithms. Motivated by a novel framework for performance evaluation on image segmentation using Pareto front, we propose a multi-metric evaluation protocol, including metrics for contour-based spatial matching, temporal consistency, user workload and time consumption. Taking the characteristic of a user-assisted video object extraction system into consideration, we formulate the metrics in a way simple but close to the assessment of human visual system. For spatial matching, we define three types of errors: sharp error, smooth error and mass error, which can precisely score an extraction result. The time consistency is introduced to evaluate the stability over time of a system. In addition, as far as a user-assisted system is concerned, the workload of users is also in our metric. Incorporating multi-metric into one 4-D fitness space, we adopt the Pareto front to find the best choice of a system with optimal parameters. The tests of our evaluation method show that the multi-metric protocol is effective.
© (2003) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Na Li, Na Li, Shipeng Li, Shipeng Li, Chun Chen, Chun Chen, } "Multimetric evaluation protocol for user-assisted video object extraction systems", Proc. SPIE 5150, Visual Communications and Image Processing 2003, (23 June 2003); doi: 10.1117/12.509857; https://doi.org/10.1117/12.509857

Back to Top