Deep convolutional neural networks (CNNs) based transfer learning is an effective tool to reduce the dependence on hand-crafted features for handling medical classification problems, which may mitigate the problem of the insufficient training caused by the limited sample size. In this study, we investigated the discrimination power of the features at different CNN levels for the task of classifying epithelial and stromal regions on digitized pathologic slides which are prepared from breast cancer tissue. We extracted the low level and high level features from four different deep CNN architectures namely, AlexNet, Places365-AlexNet, VGG, and GoogLeNet. These features are used as input to train and optimize different classifiers including support vector machine (SVM), random forest (RF), and k-nearest neighborhood (KNN). A number of 15000 regions of interest (ROIs) acquired from the public database are employed to conduct this study. The result was observed that the low-level features of AlexNet, Places365-AlexNet and VGG outperformed the high-level ones, but the situation is in the opposite direction when the GoogLeNet is applied. Moreover, the best accuracy was achieved as 89.7% by the relatively deep layer of max pool 4 of GoogLeNet. In summary, our extensive empirical evaluation may suggest that it is viable to extend the use of transfer learning to the development of high-performance detection and diagnosis systems for medical imaging tasks.