The ability to accurately predict natural backgrounds over a wide range of potential image acquisition conditions is key to wide-area change detection. The lack of robustness in pure image-based techniques has led to the development of site modeling and model-supported exploitation approaches. Instead of using multiple images to build a site model, we consider the possibility of using them directly to estimate the background of a new image for change detection. Our method uses a nonlinear mean-squared estimation technique to compute image backgrounds from multiple reference images. In comparing linear and nonlinear estimators, we find that although the performance of both improves as the number of reference images increases, the nonlinear estimator yields significantly better background estimates and is much less sensitive to registration errors between images than the linear estimator. We show that as the number of references increases, the pixels within a region become physically significant; i.e., they correspond to parts of the scene with similar physical properties. Synthetic aperture radar (SAR) background modeling and change detection results including a crosssensor change detection example in which SAR reference images are used to predict the background of an electro-optical (EO) image are presented. Significant performance gains over linear change detection (up to two orders of magnitude reduction in the false alarm rate) are demonstrated in each case over a 60-deg range of SAR sensor aspect angles.