Visual object tracking is a challenging task due to the existence of significant illumination change, appearance variation, and occlusion. We propose a structured collaborative representation-based visual tracking algorithm by using both the generative appearance model and the discriminative model. First, a structured collaborative representation model based on ℓ2 -regularized least squares is used to exploit both holistic and local information of the target. In the structured collaborative representation framework, a new over-complete dictionary containing both target and background templates is proposed to enhance the robustness of the tracking algorithm. Second, the tracking task is treated as a binary classification problem, and a Bayes classifier is trained online by the use of structured collaborative representation coefficients of positive and negative samples. Furthermore, a residual error score is constructed to improve the detective ability of the tracker. Finally, target templates are updated by combining incremental subspace learning and collaborative representation together, and background templates are subsequently updated by samples around latest results. Compared with four state-of-the-art tracking algorithms in 12 challenging video sequences, the proposed tracking algorithm demonstrates a better performance overall in terms of experimental results.