To cope with the rapid development of the real applications for infrared small targets, the researchers have tried their best to pursue more robust detection methods. At present, the contrast measure-based method has become a promising research branch. Following the framework, in this paper, a speeded-up contrast measure scheme is proposed based on the saliency detection and density clustering. First, the saliency region is segmented by saliency detection method, and then, the Multi-scale contrast calculation is carried out on it instead of traversing the whole image. Second, the target with a certain “integrity” property in spatial is exploited to distinguish the target from the isolated noises by density clustering. Finally, the targets are detected by a self-adaptation threshold. Compared with time-consuming MPCM (Multiscale Patch Contrast Map), the time cost of the speeded-up version is within a few seconds. Additional, due to the use of “clustering segmentation”, the false alarm caused by heavy noises can be restrained to a lower level. The experiments show that our method has a satisfied FASR (False alarm suppression ratio) and real-time performance compared with the state-of-art algorithms no matter in cloudy sky or sea-sky background.