This paper presents a robust real-time visual fish tracking system. The proposed visual servo framework is able to track a deformed target and maintain the target always inside the field of view. For the image processing, an efficient template matching and searching method using the mean-shift theory is developed. The robustness is achieved by appending the ratio histogram, a kernel function, and the template update to the framework when the target is deformed. Experimental results show that the presented scheme works successfully for real-time fish tracking missions. The visual tracking task can also be accomplished even when a similar object crosses over the target.