Results obtained by current foreground segmentation algorithms are often corrupted with splits and defects, making them undesirable for high level applications. We present a new segmentation algorithm that takes advantage of tracking information for segmentation. First, the moving object is tracked with a mean-shift tracker. Second, the tracking information is used to construct a tracking-based appearance model, which is then used for segmentation. Finally, a mixture model is used to cope with the tracking error. Experiments show that the proposed method provides improved results in changing settings.