Multi-object tracking in semi-crowded environment is a difficult task. Traditional methods relying only on visual cues, like appearance and simple motion prediction fails because of frequent occlusions. In this paper, a new method based on formation stability is proposed to learn the interaction information among pedestrians. whatever the relationship of a pedestrian with others is, it will remain unchanged for a while. That is, when a pedestrian is occluded, the relationship with others will stay the same after the reappearance of the pedestrian. So trajectory clips caused by occlusions can be associated together. Effectiveness of this method is validated by experiments.