One of the difficulties in semantic object tracking is to trace the object precisely as time going on. In this paper, a system for 3D semantic object motion tracking is proposed. Different form other approaches which have used regular shapes as tracked region, our system starts with a specially designed Color Image Segmentation Editor (CISE) to devise shapes that more accurately describe the region of interest (ROI) to be tracked. CISE is an integration of edge and region detection, which is based on edge-linking, split-and- merge and the energy minimization for active contour detection. An ROI is further segmented into single motion blobs by considering the constancy of the motion parameters in each blob. The tracking of each blob is based on an extended Kalman filter derived form linearization of a constraint equation satisfied by the pinhole model of a camera. The Kalman filter allows the tracker to project the uncertainties associated with the blob feature points to the next frame. Feature points extraction is done by similarity test based on optimized semantic search region. Extracted feature points serially update motion parameters. Experimental results show the different stages of the system.