Semiconductor wafer is elementary unit in semiconductor industry. In the fabrication of semiconductor wafer, surface defects such as dirties, scratches, burrs, chippings and holes may be generated which severely affect the quality of downstream production. Typical inspection of these defects mainly depends on human experts inspecting system which is time-consuming and low efficiency. With the fast development of digital imaging and processing technique, Computer vision automatic inspection method has shown vast potential for product quality test. Due to low contrast and weak context characteristics of wafer surface defects, the existing methods have difficulty to extract whole defect patterns. A novel algorithm for defect contour extraction is proposed based on multi-frame differential image summation. For each side of semiconductor wafer surfaces, multiple images are captured by high resolution digital camera. For each image the gradient is calculated using common used differential mask, and then the gradient is thinned based on edge extraction using Canny operator and smoothed using Gaussian smooth filter. All refined gradient images are added up to enhance defect features and smooth defect-free regions furtherly. Finally, the Canny operator is applied again to extract whole defect’s contour from gradient summation image. Experiments using real semiconductor wafers illustrate that the proposed algorithm can detect most of defects correctly and effectively.