Scene-based nonuniformity correction technique for Infrared focal-plane array has been widely concerned as a key technology. However, the existed algorithms are now facing two major problems that is convergence speed and ghosting artifacts. The convergence speed of original constant statistics (CS) method has been demonstrated to be more rapidly than the neural network method but how to reduce ghosting artifacts efficiently is the largest challenge. To solve the ghosting problem, the conventional methods often set a threshold to wipe off the outliers, but the threshold is difficult to choose because it changes complexly for different scene. In this paper, a novel adaptive scene-based nonuniformity correction technique is presented that performs the nonuniformity correction based on CS method. Firstly, an analysis of statistical characteristic in every pixel is taken and the cause of ghosting artifacts is discussed that the underlying distribution does not satisfy the assumptions such as symmetry. For the Gaussian distribution can not describe the statistic property for every pixel’s data, a model with mixture distribution is constructed and indicates the different distribution’s influence to generate ghosting artifacts. Then, utilizing temporal statistics of infrared image sequences the proposed method applies an alpha-trimmed mean filter to estimate detector parameters instead of the conventional mean filter. The algorithm selects the parameter of the alpha-trimmed mean estimator optimally with minimizing the sample asymptotic variance estimate. Moreover, the alpha-trimmed mean filter is designed to detect the nonsymmetry points and trim out the outlier pixels such as edges or extreme distribution. Finally, the performance of the proposed algorithm is evaluated with infrared image sequences with simulated and real fixed-pattern noise. Compared with other nonuniformity correction techniques, the proposed method inherits the superiority of the CS method that converges rapidly but is more robust and gets little ghosting artifacts. The results of the simulated and the real infrared images experiments show a significantly more reliable ability to compensate for nonuniformity and reducing ghosting artifacts effectively.