Virtual clinical trials (VCTs) were introduced as a preclinical alternative to clinical imaging trials, and for the evaluation of breast imaging systems. Realism in computer models of breast anatomy (software phantoms), critical for VCT performance, can be improved by optimizing simulation parameters based on the analysis of clinical images. We optimized the simulation to improve the realism of simulated tissue compartments, defined by the breast Cooper’s ligaments. We utilized the anonymized, previously acquired CT images of a mastectomy specimen to manually segment 205 adipose compartments. We generated 1,440 anthropomorphic breast phantoms based on octree recursive partitioning. These phantoms included variations of simulation parameters—voxel size, number of compartments, percentage of dense tissue, and shape and orientation of the compartments. We compared distributions of the compartment volumes in segmented CT images and phantoms using Kolmogrov-Smirnov (KS) distance, Kullback-Leibler (KL) divergence and a novel distance metric (based on weighted sum of distribution descriptors differences). We identified phantoms with the size distributions closest to CT images. For example, KS resulted in the phantom with 1000 compartments, ligament thickness of 0.4 mm and skin thickness of 12 mm. We applied multilevel analysis of variance (ANOVAN) to these distance measures to identify parameters that most significantly influence the simulated compartment size distribution. We have demonstrated an efficient method for the optimization of phantom parameters to achieve realistic distribution of adipose compartment size. The proposed methodology could be extended to other phantom parameters (e.g., ligaments and skin thicknesses), to further improve realism of the simulation and VCTs.