In infrared image, the grey distribution of background and target are instability, so it has much difficulty in the target segmentation. In this paper, a novel image segmentation algorithm is presented which is based on Contourlet transform and background complexity. Firstly, using Contourlet transform, the structure information of target and background is obtained. Next, structure similarity of target and background is computed. Finally, through the structure similarity of target and background, segmentation threshold is adjusted adaptively. If the structure similarity of target and background is low, it indicates that background is simple, segmentation threshold is set with the grey information. If the structure similarity of target and background is high, segmentation threshold is set with the structure information. The simulation experiments show that the target can be segmented truly in the complex background environment. The algorithm not only reserves the advantage of the grey segmentation in simple background environment, but overcomes the limitation of the grey segmentation in complex background environment, shows better adaptability than the traditional image segmentation methods.