Under pedestrian and vehicle mixed traffic conditions, the potential accident rate is high due to a complex traffic environment. In order to solve this problem, we present a real-time cognitive vision system. In the scene-capture level, foreground objects are extracted based on the combination of spatial and temporal information. Then, a coarse-to-fine algorithm is employed in tracking. After filtering-based normal tracking, problems of the target blob missing, merging, and splitting are resolved by the adaptive tracking modification method in fine tracking. For greater robustness, the key idea of our approach is adaptively adjusting the classification sensibility of each pixel by employing tracking results as feedback cues for target detection in the next frame. On the basis of the target trajectories, behavior models are evaluated according to a decision logic table in the behavior-evaluation level. The decision logic table is set based on rules of real scenes. The resulting system interprets different kinds of traffic behavior and warns in advance. Experiments show robust and accurate results of abnormality detection and forewarning under different conditions. All the experimental results run at real-time frame rates (⩾25 fps) on standard hardware. Therefore, the system is suitable for actual Intelligent Traffic System applications.