The automatic detection of defects is an essential part of the printed circuit board (PCB) production process. In recent years, while great progress has been made in the detection of PCB defects, there are still various problems in traditional defect detection methods, for example, over-reliance on the perfect template, difficult to achieve precise image registration, and highly vulnerable to environmental factors such as light, noise, and reflectivity. We propose a fast defect detection network. On one hand, this algorithm solved the problems of traditional methods. On the other hand, this algorithm solved the problems of large model size and poor real-time of existing deep learning methods. First of all, the k-means clustering algorithm is used to obtain more reasonable anchors boxes; second, an improved MobileNetV2 is used as the backbone network; after the feature extraction network, the spatial pyramid pooling (SPP) structure is introduced to increase the receptive field of the image; then, we use complete intersection over union to optimize the loss function; finally, we build an enhanced feature extraction network based on the feature pyramid network for multi-scale feature fusion. The experimental results show that this method has small model size, good real-time, and good portability, which is suitable for practical production.