Pedestrian detection is one of the most critical but challenging components in advanced driver assistance systems. Far-infrared (FIR) images are well-suited for pedestrian detection even in a dark environment. However, most current detection approaches just focus on pedestrian patterns themselves, where robust and real-time detection cannot be well achieved. We propose a fast FIR pedestrian detection approach, called MAP-HOGLBP-T, to explicitly exploit the scene context for the driver assistance system. In MAP-HOGLBP-T, three algorithms are developed to exploit the scene contextual information from roads, vehicles, and background objects of high homogeneity, and we employ the Bayesian approach to build a classifier learner which respects the scene contextual information. We also develop a multiframe approval scheme to enhance the detection performance based on spatiotemporal continuity of pedestrians. Our empirical study on real-world datasets has demonstrated the efficiency and effectiveness of the proposed method. The performance is shown to be better than that of state-of-the-art low-level feature-based approaches.