Efficient utilization of videos of lectures and presentations requires indexing based on extracted background which includes slides and / or handwritten notes. Since the background in such videos is constantly evolving there is a need for special techniques for background recovery. The objective of this paper is a method for automatically extracting the evolving background in such videos. In contrast to general background subtraction techniques which aim at extracting foreground objects, the goal here is to extract the background and complete it where the foreground is removed. Experimental results comparing the proposed approach to other known techniques demonstrate improved performance when using the proposed approach.