This paper presents a new strategy of handwritten digit recognition based on the improved VGG16 model that aims at the lack of texture features, the large differences between handwritings, and the difficulty of extracting effective information. In order to maximize the efficiency of the improved VGG16 model, an automatic drop-based learning rate scheduling is proposed to improve the SGD algorithm in the learning procedure. And automatic adaptation technique of the SGD optimizer's learning rate parameter according to the changes of accuracy rate in the previous training iterations is utilized for the training of the model, which not only apparently improves the learning results, but also speeds up the training convergence. The improved VGG16 model was evaluated on the extended MNIST dataset, achieving a high recognition accuracy rate of 99.97%. Experimental results demonstrate that the improved VGG16 model has obviously higher recognition accuracy than traditional classifiers, has stronger feature extraction ability and can meet the requirements of handwritten digits classification and recognition.