Crack is one of important damages on real concrete surface. The visual inspection that depends on inspectors, a primary method to detect cracks, is laborious and time-consuming in practical operation. Fortunately, image processing techniques make the crack detection more automated to some extent. However, the extracting of features is certainly necessary when image processing techniques detect crack in an image. As a result, the usage of image processing techniques is also limited, since images taken on real concrete surface are influenced by some noises caused by lighting, blur, and so on. In this paper, a method of convolutional neural networks-based crack detection for real concrete surface was proposed. The convolutional neural networks (CNNs) can learn the features of images automatically instead of extracting features, and therefore the CNNs will not be influenced by the noises. A convolutional neural network (CNN) used to detect crack was designed through fine-turning an existed CNN architecture. In order to train the CNN, image datasets needed be built firstly. A large number of images were taken from real concrete surface using a smartphone, cropped into small images, classified and labeled. A CNN classifier used to detect crack can be obtained by training the CNN according to those built datasets. Through integrating the trained CNN classifier into a smartphone application, the detection of crack in an image can be implemented automatically. The results illustrate that the proposed method shows high accuracy and robust performance and can indeed detect crack on real concrete surface.