In practical applications, it is urgent to develop lightweight, fast, and accurate face detectors. Although the performance of face detection has made tremendous progress with the rapid development of deep learning, they still struggle to meet the requirements of both efficiency and effectiveness. This is because current face detectors either use large networks or cumbersome processes oriented for accuracy but leading to heavy model complexity and slow inference speed or use various simple designs oriented for efficiency but greatly sacrificing accuracy or not actually efficient. In this paper, we conduct an in-depth study of the network structure and anchor setting and present an efficient and effective face detector named EEFDet, which can detect faces with lightweight model, fast inference speed, and excellent accuracy. First, a lightweight, fast, and robust network (LFRNet) is constructed by carefully designing the backbone network, detection neck, and detection head of the network. LFRNet not only has small model complexity and low latency but also possesses robust feature extraction ability. Then, a sparse anchor strategy is proposed by rationally setting anchors associated with the detection head to avoid using too many redundant anchors and get the most effective anchors. This strategy enables the detector to process multiple anchor-related parts at a fast speed while still guaranteeing detection accuracy. Extensive experiments and detailed ablation analysis are conducted on several widely used face detection benchmarks based on multiple metrics and different platforms. The results show that our EEFDet is lightweight (1.195 M parameters and 0.719 G FLOPs), fast (38.780 ms on GPU and 141.787 ms on CPU), and accurate [90.8%/89.6% (easy), 88.1%/87.2% (medium), 77.9%/77.4% (hard) APs on Val/Test set of WIDER FACE, 96.3% discrete score and 79.7% continuous score on face detection data set and benchmark (FDDB), 97.7% AP on PASCAL FACE, and 98.2% AP on annotated face in-the-wild (AFW)], achieving an optimal trade-off between efficiency and effectiveness compared to state-of-the-art face detectors.