It is always a challenging task to detect an infrared (IR) small target with high-detection rate, low false alarm rate, and high detection speed since the target usually has a small size and dim gray value, and the background is usually complex. Local contrast, which is based on the human visual system, has been proved efficient for IR small target detection, but existing local contrast algorithms are either difference-form or ratio-form and cannot enhance true target and suppress background simultaneously. In addition, most of them are pixel-level algorithms and require a huge amount of calculations. A simple but efficient method named subblock-level ratio-difference joint local contrast measure (SRDLCM) is proposed; it can enhance real small target and suppress complex background simultaneously. In addition, SRDLCM is calculated for each subblock but not each pixel, so its calculation amount can be reduced significantly. The experimental results on seven real IR sequences and one single-frame image dataset show that the proposed algorithm can achieve a good detection performance in detection rate and false alarm rate with a good robustness, and the time consumed for a single frame is only less than 0.04 s.