For naval surveillance, automatic detection of surface objects, like vessels, in a maritime environment is an
important contribution of the Electro-Optical (EO) sensor systems on board. Based on previous research using
single images, a background estimation approach using low-order polynomials is proposed for the automatic
detection of objects in a maritime environment. The polynomials are fitted to the intensity values in the image
after which the deviation between the fitted intensity values and the measured intensity values are used for detection. The research presented in this paper, includes the time information by using video streams instead of single images. Hereby, the level of fusing time information and the number of frames necessary for stable detection and tracking behaviour are analysed and discussed. The performance of the detection approach is tested on a, during the fall of 2007, collected extensive dataset of maritime pictures in the Mediterranean Sea and in the North Sea on board of an Air Defence Command frigate, HNLMS Tromp.