Recent researches on learning local descriptor are mainly based on Convolutional Neural Network (CNN), including loss function, network architecture, sample mining, etc. In this paper, we investigated the existing datasets and difficult sample mining methods for learning local feature descriptor. And we find that the samples in the train set are often not enough for various situations. Because the changes in illumination and view are continuous, but the samples are discrete. In the meanwhile, current sample mining ways only excavate existing data of train set, which is not optimal. So we proposed a new sample mining fashion, called latent hard sample mining, which can utilize potential samples for learning. This method can mitigate the influence of inadequate training samples. We compare our approach to recently introduced convolutional local feature descriptors on Photo-tour and Hpatches dataset with different losses, and demonstrate the advantages of the proposed methods in terms of performance.