The authors propose a fragment-based variational filtering technique for human tracking. Based on human classifiers and histograms of oriented gradients descriptor, more informative local parts of the human body are selected in the reference model and updated during the tracking process. Hyper-parameters of the variational Bayesian filter are adaptively tuned in order to cope with variable scenes and occlusions. To speed up the initialization and reference updating, an efficient motion cue is fused with the human detection. Extensive experimental results on benchmark datasets show that the proposed tracker is effective and robust.