It is difficult to study human actions on visual cognition as individual differences and dynamic environment causes a large number of variables. Adaptive mining the connectivity of moving human contour in infrared images based on regions can improve detecting moving object performance. We propose adaptive motion detection algorithm based on layering frequency sifting and maximally similar regions measuring in this letter, to overcome difficulties to sample moving human contour from dynamic background. First using frequency sifting layer by layer of input infrared images by Bidimensional Empirical Mode Decomposition (BEMD) representations, the original images were layered into bidimensional intrinsic mode functions (BIMFs). Thus connected edge information is remained on BIIMFs while smoothing data is filtered. Then detected connected regions using Maximally Stable Extremal Regions(MSERs) representation amongst BIMFs and the original image. Since being similarity amongst those connected regions of those images, which includes the moving human contour. At last measured similar MSERs regions hierarchically. The maximal similar connected regions segmented is candidate moving object contours. The experiment results on several open infrared videos show that the proposed algorithm improves credibility and simplicity, superior to other unsupervised measures.