The application of mosaic focal planes to the problem of surveillance has resulted in increased interest in sources of background clutter. An ideal staring sensor would be able to perfectly suppress spatial variations in the background scene. This can be accomplished; for example, by simple frame subtraction. However, any practical sensor will be limited in its suppression capability by platform motions. The result is that spatial variations are seen as temporal change. A variety of on and off focal plane background suppression algorithms have been proposed. In this paper causes of earth background clutter, particularly in the thermal infrared region are discussed. Specific examples of high clutter scenes such as: edges, anisotropic backgrounds, and inhomogeneous backgrounds are discussed both in terms of their effects on generic processors and in terms of the underlying scene physics. Examples are presented from available thermal band infrared data where possible. The scene physics based on differential insolation, material differences, heat transfer, altitude differences, as well as temperature differences is treated via the examples.