Tumor downstaging after neoadjuvant chemoradiation (CRT) in rectal cancer patients is typically assessed via Magnetic Resonance Imaging (MRI) in order to determine follow-up surgical interventions, but is associated with marked inter-reader variability and limited performance. While radiomic features have shown promise for evaluating chemoradiation response and tumor stage in rectal cancers, there is a need to determine how reproducible these features are across different MRI scanners and acquisitions. In this study, we evaluated radiomic feature reproducibility in terms of feature instability within a uniquely curated true healthy" rectum cohort in order to construct a stability-informed radiomic classifier for differentiating poorly from markedly down-staged rectal tumors after chemoradiation in a multi-site setting. We utilized a cohort of 156 patients, with (a) 74 MRIs visualizing the healthy rectum, (b) 52 post-CRT MRI scans in the discovery cohort, and (c) 30 post-CRT MRI scans in a second-site validation cohort; the latter 2 being from rectal cancer patients. 764 radiomic features were extracted from within the entire rectal wall on each MRI scan. Feature instability was used to quantify how reproducible each radiomic feature was between the discovery cohort and the healthy rectum cohort, using locations along the rectum that were spatially distinct from the treated tumor region. From the resulting stability-informed" feature set, the most relevant features were identified to distinguish pathologic tumor stage groups in the discovery cohort via a QDA classifier with cross-validation to ensure robustness. The top 4 radiomic features were then evaluated in hold-out fashion on scans from the validation cohort. We found that utilizing a stability-informed radiomic model (which comprised features that were reproducible in 100% of all comparisons) was significantly more accurate in identifying pathological tumor stage regression in both discovery (AUC=0:66 ± 0:09) and validation (AUC=0.73) cohorts, compared to a basic radiomic model that used all extracted features (AUC=0:60 ± 0:07 in discovery, AUC=0.62 in validation). Evaluating feature instability with respect to healthy rectal tissue may thus enhance the performance of radiomic models in characterizing pathologic downstaging in rectal cancers, via MRI.