The pose issue which may cause loss of useful information has always been a bottleneck in face and ear recognition. To address this problem, we propose a multimodal recognition approach based on face and ear using local feature, which is robust to large facial pose variations in the unconstrained scene. Deep learning method is used for facial pose estimation, and the method of a well-trained Faster R-CNN is used to detect and segment the region of face and ear. Then we propose a weighted region-based recognition method to deal with the local feature. The proposed method has achieved state-of-the-art recognition performance especially when the images are affected by pose variations and random occlusion in unconstrained scene.
Descriptor is the key of any image-based recognition algorithm. For ear recognition, conventional descriptors are either based on 2D data or 3D data. 2D images provide rich texture information and human ear is a 3D surface that could offer shape information. It also inspires us that 2D data is more robust against occlusion while 3D data shows more robustness against illumination variation and pose variation. In this paper, we introduce a novel Texture and Depth Scale Invariant Feature Transform (TDSIFT) descriptor to encode 2D and 3D local features for ear recognition. Compared to the original Scale Invariant Feature Transform (SIFT) descriptor, the proposed TDSIFT shows its superiority by fusing 2D local information and 3D local information. Firstly, keypoints are detected and described on texture images. Then, 3D information of the keypoints located on the corresponding depth images is added to form the TDSIFT descriptor. Finally, a local feature based classification algorithm is adopted to identify ear samples by TDSIFT. Experimental results on a benchmark dataset demonstrate the feasibility and effectiveness of our proposed descriptor. The rank-1 recognition rate achieved on a gallery of 415 persons is 95.9% and the time involved in the computation is satisfactory compared to state-of-the-art methods.