We investigated the relationship between the face recognition performance of individuals and their eye movement characteristics that were measured while each subject observed the faces that were displayed on a screen. We formulated the statistical nature of their eye movements from a machine-learning perspective by applying a hidden Markov model (HMM). We used a set of computer-generated faces that included both the images of actual faces and synthetic images obtained by slightly transforming the impressions of the original faces. With these visual stimuli, we conducted a simple face recognition experiment, and subjects judged whether they had seen the faces before. We obtained a quantitative hit rate score for each stimulus and subject. We also tracked their eye movements and recorded as temporal chains their eye fixation points using an eye-tracking system. For each class of face stimulus and subject, we estimated the HMM parameters from the training samples of the eye movement. For the given eye movement data as test samples, we conducted a classification test among the pre-defined classes based on the differences of the log-likelihood values obtained from each HMM. Better discrimination of the subjects by the HMM-based classification of the eye movement data corresponded to lower face recognition scores by the subjects, suggesting that individually consistent eye movement patterns may lower the face recognition performance by humans.