Background prior selection is the key step in current ranking-based saliency detection approaches. The existing related methods usually choose boundary regions of an image or the region with low initial saliency value in single image scale as the background. Then, the saliency map is obtained by ranking the inside similarity and correlation. However, these methods cannot handle situations in which the salient object lies on the image boundary (boundary-salient) multiple salient objects exist in a single image (multisalient). To this end, this paper proposes an adaptive background selection method by exploiting the background invariance in different image scales within distinct color spaces. Through embedding the selected background prior into multiple newly proposed ranking-based saliency methods, the superiority of the obtained background prior is strongly verified. Exhaustive experiments on four challenging datasets demonstrate that the proposed method outperforms the state-of-the-art methods in handling the boundary-salient and multisalient situations.