We present an abandoned luggage detection architecture that consists of an intelligent surveillance system for public places. Detection of abandoned luggage is necessary because unattended or abandoned luggage can be used as a means of terrorist attack, especially for bombs. Our proposed system relies on three modules: moving object detection, object tracking and classification, and event recognition. We focus on abandoned luggage detection. To recognize an abandoned luggage event, we constructed the finite state automaton (FSA), in which each FSA state represents a certain luggage status. The proposed algorithm shows good performance in a real-world environment and also works at real-time speed.