This paper presents an adaptive part-based probabilistic model for non-rigid object tracking. Without any assumption on scenes or poses, our model is online generated and updated. The parts in the model are extracted by clustering based on the appearance consistency of local feature descriptors in the object. A probability indicating the possibility of a part belonging to the object is then assigned to each part and adapted during tracking. We also propose a fully automatic algorithm for single object tracking with model matching and adaption. Our approach is evaluated on three different datasets and compared with previous work on visual tracking. The experimental results showed that our approach can track non-rigid object under occlusion and object deformation effectively in real time. Moreover, it works even if the target is partially occluded at initialization step.