We propose a foreground extraction method from a freely moving RGB-D video camera that uses a nonlinear optimization function to classify pixels in every frame as either foreground or background. The method starts with extracting sparse features in every frame. These features are then matched to create a nonlinear optimization function that is used to label each pixel in the depth image as either foreground or background. Finally, the inverse mapping from depth to RGB data is used to extract the foreground from the RGB image. Our results show that even in the presence of high camera motion, it is possible to reliably extract the foreground from the RGB-D video data captured from a freely moving camera.