Tracking objects with the scale variations quickly is a challenging problem in visual tracking. Most existing methods estimate the scale of the object with an exhaustive search strategy, which needs large calculations but with less improvement. We propose to use twofold weighted phase correlation with the kernel (WPCK) to simultaneously estimate the translation and scale for the object during tracking. Besides, from the linear phase correlation, a weighted kernel phase correlator is first introduced to improve tracking performance. It extends the filter into a nonlinear space, which is more robust to signal noises and distortions. To efficiently locate the object in a large three-dimensional space for real-time running, we formulate tracking problem into two independent subproblems: the translation offset and scale offset. The mechanism is: through the log-polar transform of the amplitude spectrum, the scale estimation can be performed with phase correlation, which is also utilized to solve the translation offset. Thus, the accurate object status can be achieved effectively by twofold WPCK. Extensive experimental results on OTB2013, OTB100, and UAV123 datasets reveal that the proposed method has superior performance gains over several state-of-the-art trackers in terms of accuracy, robustness, and speed.