Surface defect recognition is one of the key technologies for varistor quality inspection, which can greatly improve detection efficiency and performance. In order to more accurately identify the surface defects of a varistor body and the pins, a method for identifying the surface defects based on deep convolutional neural networks (CNN) is proposed. The proposed method mainly includes four stages: image acquisition and data set construction, convolutional neural network modeling, CNN training and testing. Firstly, varistor images are acquired, and the body and pins of the varistor are segmented by image segmentation method. The number of samples is increased by data augmentation to make a data set of 5 classes. Secondly, according to the appearance characteristics of varistor, a CNN model is designed for varistor surface defect recognition. Third, using the created data set, the training data set with category labels are input to the proposed CNN for training. Finally, 1200 test samples were tested on the trained model in the test phase and the performance of the proposed algorithm was evaluated using mean average precision. The experimental results show that our method can identify the surface defects of the main body and pins of varistor efficiently and accurately.