An image registration quality evaluator (RQE) is proposed to automatically quantify the accuracy of registrations. The RQE, based on an adaptive pattern classifier, is generated from a pair of reference and target images. It is unique to each patient, anatomical site and imaging modality. RQE is applied to patient positioning in cranial radiotherapy using portal/portal and portal/DRR registrations. We adopted 1mm translation and 1° rotation as the maximal acceptable registration errors, reflecting typical clinical setup tolerances. RQE is used to determine the acceptability of a registration. The performance of RQE was evaluated using phantom images containing radio-opaque fiducial markers. Using receiver operating characteristic (ROC) analysis, we estimated the sensitivity and the specificity of the RQE are 0.95 (with 0.89-0.98 confidence interval (CI) at 95% significance level) and 0.95 (with 0.88-0.98 CI at 95% significance level) respectively for intramodal RQE. For intermodal RQE, the sensitivity and the specificity are 0.92 (with 0.81-0.98 CI at 95% significance level) and 0.98 (with 0.89-0.99 CI at 95% significance level) respectively. Clinical use of RQE could significantly reduce the involvement of the oncologist for routine pre-treatment patient positioning verification, while increasing setup accuracy.