Current keypoint-based trackers are widely used in object tracking system because of their robust capability against scale, rotation and so on. However, when these methods are applied in tracking 3D target in a forward-looking image sequences, the tracked point usually shifts away from the correct position as time increases. In this paper, to overcome the tracked point drifting, structured output tracking is used to track the target point with its surrounding information based on Haar-like features. First, around the tracked point in the last frame, a local patch is cropped to extract Haar-like features. Second, using a structured output SVM framework, a prediction function is learned in a larger radius to directly estimate the patch transformation between frames. Finally, during tracking the prediction function is applied to search the best location in a new frame. In order to achieve the robust tracking in real time, keypoint matching is adopted to coarsely locate the searched field in the whole image before using the structured output tracking. Experimentally, we show that our algorithm is able to outperform state-of-the-art keypoint-based trackers.