Evaluating tumor regression of rectal cancers via MRI after standard-of-care chemoradiation therapy (CRT) remains highly challenging for radiologists. While the tumor region-of-interest (ROI) on post-CRT rectal MRI is difficult to localize, an underexplored region is the perirectal fat (surrounding tumor and rectum) where residual cancer cells and positive lymph nodes are known to be present. Recent studies have shown that physiologic environments surrounding tumor regions may provide complementary information that is predictive of response to CRT and patient survival. We present initial results of characterizing perirectal fat regions on MRI via radiomics, towards capturing sub-visual details related to rectal tumor or nodal response to CRT. A total of 37 rectal cancer patients for whom MRIs as well as pathologic tumor staging were available post-CRT were included in this study. Region-wise radiomic features were extracted from expert annotated perirectal fat regions and a 2-stage feature selection was employed to identify the most relevant features. Radiomic entropy of perirectal fat was found to be over-expressed in patients with poor tumor or nodal response post-CRT, albeit with different spatial distributions. In a leave-one-patient-out cross validation setting, a quadratic discriminant analysis (QDA) classifier trained on top radiomic features from the perirectal fat achieved AUCs of 0.77 (for differentiating incomplete vs marked tumor regression) and 0.75 (for differentiating lymph node positive from negative patients). By comparison, perirectal fat intensities achieved significantly poorer AUCs in both tasks. Our results indicate perirectal fat on post-CRT MRI may be highly relevant for evaluating CRT response and informing follow-on interventions in rectal cancers.