An elliptic face segmentation algorithm, called a facial component extractor (FCExtractor), was recently proposed. The algorithm is based on a novel overcomplete wavelet template, a support vector machine (SVM) classifier, and wavelet entropy filtering. It is designed to consistently detect and segment the eyes-nose-mouth T-shaped facial region via ellipse. Thereafter, head orientation is estimated by using the ratio of cheeks. To evaluate the effectiveness of the FCExtractor, we introduce a face detection measure based on the distance between the expected and segmented eye-mouth triangle circumscribed circle areas. We then apply the local description of the segmented face through normalization, illumination normalization, log-polar mapping, and self-eigenface to achieve recognition. The novelty of this approach for face representation comes from the derivation of the likelihood fitness function for self-eigenface selection of a discriminative subset and the adaptive threshold value. The approach maximizes the differences amidst face images of different persons, and it also minimizes the expression and pose variations of the same person. Experimental results on available databases and a live sequence show that our method is superior to conventional methods based on rectangular face segmentation against complex scenes.