The large volume of vehicles on the road has created new challenges for agencies responsible for law enforcement and public safety. Such agencies utilize visual surveillance technology to assist monitoring of vehicles from a remote location. These surveillance systems typically require trained human operators. Consequently, they are prone to human errors due to fatigue or diverted attention caused by excess information. Thus a need exists for an automated system that can analyze the surveillance videos and extract important information. This information would be used to detect occurrence of “anomalous” events. We propose a visual surveillance system designed to function in the above-mentioned manner. The system observes vehicular traffic from a standoff range and extracts information about the vehicles. This information includes vehicle type, make, tire size, and its trajectory. Based on this information, the system checks for anomalies in the vehicles’ appearance and/or motion. We describe analysis methods for obtaining the vehicle information from two cameras placed in an orthogonal configuration and for classifying the vehicles using these observations. We present the results of applying these methods on traffic videos. Our proposed system can be deployed for traffic monitoring (on highways/intersections) or infrastructure protection (at check points).