Video-based rapid traffic accident detection is very important for intelligent transport systems. Traditional methods are either not fast enough or not stable with working environments. A rapid and environment-adaptive method is proposed. The inspiration of the method is originated from the key observation that the traffic accident brings abundant information on motion directions. This method includes three steps. First, the orientation map for each video frame is constructed based on the optical flows. Then, for each orientation map, the connected regions are formed. An entropy-like energy function is used to measure the orientation information of the connected region. The higher the energy value, the more moving directions exist. The highest measure of these connected regions in each orientation map is considered as its energy measure. Finally, based on the energy sequence of orientation maps, a Gaussian model is established to learn the normal energy fluctuation range for some environment. In the detection process, if the energy of one orientation map burst out of the normal range, it means there exists a traffic accident. The advantages of our method include the fast processing speed, a compact parameter set, and the robustness to the different environments and illuminations. Experimental results confirm the above advantages of the proposed approach.