Polygenic risk scores (PRSs) may be used to investigate the effects of genetic risk for disease on complex human traits. Here we set out to determine how the overall genetic risk for Alzheimer’s disease (AD) shapes brain structure in non-AD populations. PRS scores were computed using results from the International Genomics of Alzheimer's Project (IGAP) study. PRSs were computed at 14 different significance thresholds in the IGAP results. The effect of PRS as a predictor of brain morphometry was mapped voxelwise on brain structure as determined by tensor-based morphometry (TBM) in three cohorts: ADNI1, ADNI2, and HUNT. Our multi-cohort TBM framework first tests associations in each cohort individually, then meta-analyzes findings in a common space. Higher PRS for AD was associated with greater ventricular and lower hippocampal volumes. Associations remained after removing the major AD risk gene, APOE, from the PRS. This cumulative influence of common genetic variants on brain-wide structural variation in nondemented individuals may pinpoint genetic and neurological pathways that contribute to the preclinical assessment of disease risk.