Tracking non-rigid objects such as people in video sequences is a daunting task due to computational complexity and unpredictable environment. The analysis and interpretation of video sequence containing moving, deformable objects have been an active research areas including video tracking, computer vision, and pattern recognition. In this paper we propose a robust, model-based, real-time system to cope with background clutter and occlusion. The proposed algorithm consists of following four steps: (i) localization of an object-of-interest by analyzing four directional motions, (ii) region tracker for tracking moving region detected by the motion detector, (iii) update of training sets using the Smart Snake Algorithm (SSA) without preprocessing, (iv) active shape model-based tracking in region information. The major contribution this work lies in the integration for a completed system, which covers from image processing to tracking algorithms. The approach of combining multiple algorithms succeeds in overcoming fundamental limitations of tracking and at the same time realizes real time implementation. Experimental results show that the proposed algorithm can track people under various environment in real-time. The proposed system has potential uses in the area of surveillance, sape analysis, and model-based coding, to name of few.