The delayed brittle fracture of high-strength bolts in long-span steel bridges threatens the safety of the bridges and even lead to serious accidents. Currently, human periodic inspection, the most commonly applied detection method for this kind of high-strength bolts damage, is a dangerous process and consumes plenty of manpower and time. To detect the damage fast and automatically, a visual inspection approach based on deep learning is proposed. YOLOv3, an object detection algorithm based on convolution neural network (CNN), is introduced due to its good performance for the detection of small objects. First, a dataset including 500 images labeled for damage is developed. Then, the YOLOv3 neural network model is trained by using the dataset, and the capability of the trained model is verified by using 2 new damage images. The feasibility of the proposed detection method has been demonstrated by the experimental results.