Object tracking is a common vision task to find and follow moving objects between consecutive frames. It has widespread applications in fields ranging from video coding, visual surveillance, and human computer interaction, to intelligent robotics.
Among numerous object tracking algorithms, mean shift (MS) object tracking has recently received growing interest since it was introduced by Comaniciu, Ramesh, and Meer.1 This method tracks an object region represented by a spatially weighted intensity histogram. An object function that compares target and candidate kernel densities is formulated using the Bhattacharyya coefficient, and tracking is achieved by optimizing this objective function using the iterative MS algorithm. Though the MS object tracking algorithm performs well on sequences with relatively small object displacement, its performance is not guaranteed when the objects move fast or undergo partial or full occlusion.
To overcome this disadvantage of the MS tracking method, an improved MS object tracking algorithm was proposed in Ref. 2 by initializing MS with the predicted value of a Kalman filter (KF). In Ref. 3, the exact target center is obtained by combining the two estimated target centers obtained by the KF and MS algorithm respectively at each frame. Since the prediction and measurement errors of KF are set as constant, this algorithm is not robust enough. A new object tracking scheme is proposed in Ref. 4 that combines the sum-of-squared-differences object tracking method and MS object tracking method in the KF framework. In this method, to handle partial occlusion, the whole object is represented by a number of elementary MS modules embedded within the object, rather than a single global MS tracker. Therefore, this scheme is time consuming.
In this work, a novel object tracking algorithm based on MS and KF is proposed. First, the system model of KF is constructed, and the center of the object predicted by KF is used as the initial value of the MS algorithm. Then the searching result of MS is fed back as the measurement of KF, and the estimated parameters of KF are adjusted by the Bhattacharyya coefficient adaptively. The proposed algorithm can accurately capture the object’s position when the object undergoes large displacements or occlusion.
The remainder of this work is organized as follows. In Sec. 2, we review MS object tracking briefly. The proposed object tracking algorithm is presented in Sec. 3. Experimental results are given in Sec. 4, followed by conclusions in Sec. 5.
Mean Shift Object Tracking
In the MS object tracking method,1 the target model is defined as its normalized color histogram , where is the number of bins. The normalized color distribution of a target candidate centered at in the current frame can be calculated asare the pixel locations of the target candidate in the target area, is the Kronecker delta function, associates the pixel to the histogram bin, is the kernel profile with bandwidth , and is a normalization constant. The same equations are used to obtain the color distribution of the target model .
The Bhattacharyya coefficient, which evaluates the similarity of the target model and the target candidate model, is defined as
To find the location corresponding to the target in the current frame, the Bhattacharyya coefficient in Eq. 2 should be maximized as a function of , which can be solved by running the MS iterations. We assume that the search for the new target location in the current frame starts at the location . At each step of the iterative process, the estimated target moves from to the new location , defined as. More information on MS object tracking can be found in Ref. 1.
Adaptive Kalman Filter for Object Tracking
In this work, the MS object tracking method is integrated into the KF framework, and an adaptive KF algorithm for object tracking is proposed. First, MS initialized by the predicted value of KF is used to search the target position. Then the searching result of MS is fed back as the measurement of KF, and the estimated parameters of KF are adjusted by the Bhattacharyya coefficient adaptively. For faster implementation, two independent trackers of KF were defined for horizontal and vertical movement.
Model of the Kalman Filter
We define the variable as the discrete time , state vector , measurement vector , state transition matrix , measurement matrix , state noise , and measurement noise . The system is expressed as:
We assume that and are Gaussian random variable with zero mean, so their probability density functions are and , where the covariance matrix and are referred to as the transition noise covariance matrix and measurement noise covariance matrix.
We design a model to track object (the details are as follows). The state vector is , where , and represent the (horizontal or vertical) center, velocity, and acceleration, respectively. The measurement vector is . The state transition matrix isis the time interval. The measurement matrix is . The transition noise covariance matrix is . The estimate of parameters and is described in Sec. 3.2.
Adaptive Kalman Filter
In the KF algorithm, the measurement error covariance and Kalman gain are in inverse ratio. As the covariance matrix approachs zero, the Kalman gain weights the residual more heavily. In this case, the measurement is trusted more and more, while the predicted result is trusted less and less. On the other hand, as the a-priori estimate error covariance of KF approaches zero, the Kalman gain weights the residual less heavily. The actual measurement is trusted less and less, while the predicted result is trusted more and more.5 Therefore, the system will achieve a near optimal result if we can decide which one to trust. In this work, the so-called adaptive KF allows the estimated parameters and of KF to adjust automatically according to the Bhattacharyya coefficient of MS object tracking.
In the MS object tracking method, the Bhattacharyya coefficient evaluates the similarity of the target and candidate models. When the tracked object is occluded by other objects or background, the Bhattacharyya coefficient will descend dramatically. Thus, we define a threshold to determine whether the occlusion happens or not.
Assuming the searching result of MS is in the current frame , the Bhattacharyya coefficient evaluates the similarity of the target model and the candidate model centered at . Since the search result of MS is used as a measurement of KF, in a correction step the Bhattacharyya coefficient is used to adjust the estimate parameters of adaptive KF. If the Bhattacharyya coefficients is more than the threshold , then the value of is set as , and is . Otherwise, it is reasonable to let and be zero and infinity, respectively, thus the Kalman gain is a zero value. To smooth temporal variations, the parameters associated with the current frame are obtained through temporal filtering,is a large constant, so the posteriori estimate of KF approximates to its predicted value, and is the forgetting factor. The lower is, the faster the update of and becomes.
According to the Bhattacharyya coefficient, the KF system can be adjusted automatically to estimate the center of the tracked object. For the sake of clarity, we present here the whole algorithm.
Input: state vector of the target’s horizontal center; state vector of the target’s vertical center and the target model .
Step 1: predict the target’s horizontal center and vertical center by using the state equation of KF, respectively.
Step 2: employ MS initialized by the predicted value of KF to search the center of the object in the current frame , then get the search results .
Step 3: compute the Bhattacharyya coefficient .
Step 5: using and as the measurements of two KFs, compute and by the correction step of KF, respectively.
To demonstrate the robustness and validity of the proposed algorithm, we describe the experiment results on real-life tracking scenarios, and compare the tracking results of the proposed algorithm with the MS object tracking algorithm and the typical KF algorithm. In the typical KF algorithm, the system model of KF is the same as in Sec. 3.1. Both the prediction and measurement errors are set as constant. They are given as 0.8 and 0.2 experimentally. In the experiment, the RGB color space was taken as feature space, and it was quantized into bins. We chose the parameters , , and experimentally. The Epanechnikov profile is used for histogram computations.
The test video sequence has 140 frames of . The results of frames 17, 105, 120, and 140 are shown in Fig. 1. The target was initialized with a hand-drawn elliptical region of size . When the person walks slowly, the proposed algorithm, MS, and typical KF algorithm can accurately capture the target’s position. At frame 79, the person begins to increase his velocity suddenly, and the MS algorithm lost the target completely at frame 105. From frame 117 to 129, the person is occluded by a tree. The MS and typical KF algorithm fail after full occlusion, whereas the proposed algorithm accurately captures the target. The Bhattacharyya coefficient values in the proposed algorithm are shown in Fig. 2. It can be seen that the Bhattacharyya coefficient values descend dramatically when the person is occluded by a tree.
In this work, the MS object tracking method is integrated into the KF framework and an adaptive KF algorithm is proposed. First, MS initialized by the predicted value of KF is used to track the target position. Then the tracking result of MS is fed back as the measurement of KF, and the estimate parameters of KF are adjusted by the Bhattacharyya coefficient adaptively. According to the Bhattacharyya coefficient, the KF can be adjusted automatically to estimate the center of the tracked object. The experimental results demonstrate the robustness and validity of the proposed algorithm.