Visual tracking is a fundamental task and has many applications in computer vision. We incorporate local dictionary and L0 regularized low-rank features into the particle filter framework to address this problem. Specifically, by developing an efficient L0 regularized sparse coding model to incrementally learn low-rank features for the tracking target and incorporating a local dictionary into low-rank features to build the observation model, we establish a robust online object tracking system. As a nontrivial byproduct, we also develop numerical algorithms to efficiently solve the resulting nonconvex optimization problems. Compared with conventional methods, which often directly use corrupted observations to form the dictionary, our low-rank feature-based dictionary successfully removes occlusions and exactly represents the intrinsic structure of the object. Furthermore, in contrast to the traditional holistic methods, the local strategy contains abundant partial and spatial information, thus enhancing the discrimination of our observation model. More importantly, the L0 norm-based hard sparse coding can successfully reduce the redundant information while preserving the intrinsic low-rank features of the target object, leading to a better appearance subspace updating scheme. Experimental results on challenging sequences show that our method consistently outperforms several state-of-the-art methods.