Cascaded face detectors using convolutional neural network (CNN) are widely adopted in real-world applications due to their superior efficiency. However, the performance of those detectors degrades significantly with the presence of challenging faces. In this paper, inspired by the fact that large receptive field provides more contextual information for improved performance, based on the existing cascaded CNN face detector, we propose a context-aware cascaded face detector, which aggregates more information for each candidate window. We empirically evaluated different methods for enhancing the information embedded in the receptive field and provide detailed comparisons. Moreover, we propose a score fusion strategy to combine the confidence scores from different detecting stages in the cascaded network to get a more reliable confidence score. We also propose an improved strategy to perform non-maximum suppression that achieves better efficiency when the number of proposals is huge. Extensive experiments on FDDB and WIDER FACE datasets demonstrate that our real-time detector achieves better performance over other cascaded detectors.