Recently, tracking is regarded as a binary classification problem by discriminative tracking methods. However, such binary classification may not fully handle the outliers, which may cause drifting. We argue that tracking may be regarded as one-class problem, which avoids gathering limited negative samples for background description. Inspired by the fact the positive feature space generated by one-class support vector machine (SVM) is bounded by a closed hyper sphere, we propose a tracking method utilizing one-class SVMs that adopt histograms of oriented gradient and 2bit binary patterns as features. Thus, it is called the one-class SVM tracker (OCST). Simultaneously, an efficient initialization and online updating scheme is proposed. Extensive experimental results prove that OCST outperforms some state-of-the-art discriminative tracking methods that tackle the problem using binary classifiers on providing accurate tracking and alleviating serious drifting.