Hand tracking is becoming more and more popular in the field of human-computer interaction (HCI). A lot of studies in this area have made good progress. However, robust hand tracking is still difficult in long-term. On-line learning technology has great potential in terms of tracking for its strong adaptive learning ability. To address the problem we combined an on-line learning technology called on-line boosting with an off-line trained detector to track the hand. The contributions of this paper are: 1) we propose a learning method with an off-line model to solve the drift of on-line learning; 2) we build a framework for hand tracking based on the learning method. The experiments show that compared with other three methods, the proposed tracker is more robust in the strain case.