In both the public and private sectors, there is increasing demand for systems that take advantage of on-line sensor data and especially real-time video data. In many of these applications, special-purpose sensors, coupled with structured work environments, have made it possible to deploy versions of working systems. Despite these early successes, however, the economics of deployment remain unfavorable. In the security and surveillance domain, the ability to do quick detection and classification of objects can add value to systems monitoring an interior area for intruders or performing outdoor perimeter control. Current-generation motion-detection systems are hampered by their inability to recognize or classify the objects causing the motion. The coupled detection, tracking, and recognition techniques reported in this paper would immediately increase the value of intrusion-detection systems and reduce the personnel needed to man them. This paper reports on progress in three areas: The first emphasizes real-time perception mechanisms involving motion based detection and tracking of figures with a camera having pan-tilt-zoom control. The second component focuses on recognition mechanisms which allow reacquisition of tracked targets when they undergo various kinds of occlusion. It will also be used for discriminating people from other moving agents. The third discusses trends in real-time system performance arguing that the gap between commodity PC platforms and special purpose accelerators is narrowing.