In this paper, we propose a stable illumination change adaptive tracking algorithm in which the color model update and the target localization are mutually constrained to each other. The mutual constraint is the result of sharing the same five-dimensional feature space, where the feature vector is composed of the three color components and the x,y coordinates of pixels inside the target region. The use of the five-dimensional feature vector introduces spatiality in the color model update, i.e., the re-clustering of the feature space is constrained by the spatial location of the pixels corresponding to the target color. The spatial constraint on the color model update is also due to the use of a window which contains the pixels used in the color model update, where the location of the window is decided by the current target location. The color model update is performed by a five-dimensional mean shift-based clustering algorithm. The update in the color components in the five-dimensional feature vector handles the illumination change in the colors in the target region. The five-dimensional mean shift-based clustering updates also the x,y coordinates in the five-dimensional vector, and thus constrains the target localization. The target localization uses the x,y coordinates of the mean vectors of the five-dimensional vectors corresponding to each color bin. The target location is computed as a minimizer of a proposed energy functional, which is proposed such that the relative spatial locations of the mean vectors are considered in the target localization. The mutual constraint of the color model update and the target localization makes the tracking stable under various situations such as global illumination change, partial occlusion, and cluttered background. It can stably track the target even in the appearance of major background colors that are also major colors of the target object.