We present a new solution to the known problem of video keying in a natural environment. We segment foreground objects from background objects using their relative distance from the camera, which makes it possible to do away with the use of color for keying.
To do so, we developed and built a novel depth video camera, capable of producing RGB and D signals, where D stands for the distance to each pixel. The new RGBD camera enables the creation of a whole new gallery of effects and applications such as multi-layer background substitutions. This new modality makes the production of real time mixed reality video possible, as well as post-production manipulation of recorded video.
We address the problem of color spill -- in which the color of the foreground object is mixed, along its boundary, with the background color. This problem prevents an accurate separation of the foreground object from its background, and it is most visible when compositing the foreground objects to a new background.
Most existing techniques are limited to the use of a constant background color. We offer a novel general approach to the problem with enabling the use of the natural background, based upon the D channel generated by the camera.