The sensitivity of modern infrared sensors, using cooled Focal Plane Arrays (FPA's), allows the detection of small incoming airborne targets at sea as soon as they pop-up over the horizon. Experiments show strong fluctuations of the contrast-irradiance level of target and background at the entrance pupil of the sensor. The characteristics of these fluctuations have an impact on the design of detection- and tracking algorithms of the Infrared Search and Track (IRST) sensors and tracking systems. Several physical phenomena may contribute to the apparent contrast intensity (and thus signal) variations, such as specular sun- or cold sky reflections at the target surface, changes in the aspect angle and height of the target, coherence of the target radiation dependent on its size and range, atmospheric turbulence, wave chopping, diffraction at the wave tops, wave effects on the temperature profile in the atmospheric boundary layer, local droplet clouds produced by breaking waves, detector temporal- and spatial noise and under-sampling and fill-factor effects when the sensor is panning over the scene. The magnitude of each of these effects is estimated and compared to data from measurements. It is in general not simple to determine which effect is dominant in a certain case. The low frequency behaviour of the signal variations suggests that "classical" scintillation is not the major contributor in most cases. It is shown that structure variations in the boundary layer along the path due to waves, may be responsible for most of the low frequency effects.