In this paper, we present a technique for object tracking in image sequences, which makes use of an active contour framework that moves vertices of a polygon. In this approach, upon capturing boundaries of an object by a polygon from the first few frames of the image sequence, both the spatial segmentation and motion segmentation of a polygonal object can be achieved quickly by involving only the vertex locations and the adjacent edges in the computations. We carry out velocity field estimation at an active polygon vertex using the optical flow constraint on its two adjacent edges. A spatial segmentation phase follows to further refine object’s vertex locations estimated by the optical flow. The advantage of our region-based active polygons over continuous active contours in object tracking in video applications is highlighted by its provision of a compact representation of object features, particularly for simply connected target shapes, hence will be essential for their tracking. Application of the method to target tracking in IR image sequences is illustrated.