Due to various reasons (e.g., poor quality of image, sudden changes in light conditions, occlusions, or objects with similar colors), holes may appear in the moving object region when performing moving object segmentation. How to remove these holes is a crucial problem in high-quality segmentation. A moving object segmentation approach combining symmetrical differencing and joint bilateral filtering is proposed. One important aspect of our approach is that joint bilateral filtering is used to eliminate/reduce the holes in the moving object region, which makes the extracted moving object more accurate and complete, in many cases in contrast to the reference temporal difference approaches. The approach first performs block-based symmetrical differencing to obtain coarse moving object regions. Then, a joint bilateral filter that uses the current gray image as a guide image is adopted to smooth the difference image. Next, edge detection is performed on the smoothed difference image and the current gray image, respectively, to get the edges of moving objects. Finally, the undesired background is distinguished from the moving object and is cut off by a postprocessing module. In addition, the fast joint bilateral filters that can be used as substitutes for the classical joint bilateral filter are discussed. Experimental results show that this approach can effectively fill the holes in the moving object region and provide an improvement in accuracy of moving object segmentation in many cases.