This paper proposes a semiautomatic algorithm for the accurate extraction of an athlete from color diving sequences. Change detection techniques and edge detection techniques are combined to extract the moving object. Color information and interactive information are used to get rough region of the athlete interested. A robust edge map is derived from the difference between successive frames, then further refining of rough athlete region is applied by the information of the robust edge. The proposed method is useful in applications with a relatively still background, Experimental results show that the method provides accurate extraction with pixel-wise precision, thus providing a reliable input to further analysis or applications such as MPEG-4.