Representing an object with multiple image fragments or patches for target tracking in a video has proved to be able to maintain the spatial information. The major challenges in visual tracking are effectiveness and robustness. We propose a robust fragments-based tracking algorithm with adaptive feature selection. The best discriminate feature is used for tracking, which can improve tracking effectiveness. A set of likelihood images corresponding to the most discriminative features are fused to divide the object into some fragments, which can maintain the spatial information. By weighting the fragment and background colors, more robust target and candidate models are built. Given these advantages, the novel tracking algorithm can provide more accurate performance and can be directly extended to a multiple object-tracking system.