In fine-grained object recognition task, over-fitting problem often occurred due to the small number of fine-gained data for each category, especially for the CNN with deep layers and millions parameters. Therefore, we proposed a data augmentation method based on interest points of feature, which can alleviate the over-fitting problem and improve the classification accuracy effectively. The key idea of our method is finding the interest points that attract the classifier which come from the output of the CNN middle layer, locating the areas corresponding to the interest points in the original images and cutting the areas out for augmentation. All work can be done through training procedure completely. The method requires no additional training models and more other parameters. We applied the proposed data augmentation method on CUB200-2011, Stanford Dogs and Aircraft datasets and achieved excellent performance with 11.32% classification accuracy improvement. The experiment results showed that the proposed method can mitigate the problem of over-fitting in fine-grained images.