Here, we describe a novel approach for small surface object detection with an onboard infrared (IR) camera working in maritime scenes. First, we propose a simple but effective tool called the local minimum patterns (LMP), which are theoretically the approximated coefficients of some stationary wavelet transforms, for single image background estimation. Second, potential objects are segmented by an adaptive threshold estimated from the saliency map, which is obtained by background subtraction. Using the LMP based wavelet transforms and the histogram of the saliency map, the threshold can be automatically determined by singularity analysis. Next, we localize potential objects by our proposed fast clustering algorithm, which, compared with popular K-Means, is much faster and less sensitive to noises. To make the surveillance system more reliable, we finally discuss how to integrate multiple cues, such as scene geometry constraints and spatio-temporal context, into detections by Bayesian inference. The proposed method has shown to be both effective and efficient by our extensive experiments on some challenging data sets with a competitive performance over some state-of-the-art techniques.