We have developed a simulation tool for modeling the performance of CMOS-based medical x-ray detectors, based on the Monte Carlo toolkit GEANT4. Following the Fujita-Lubberts-Swank approach recently reported by Star-Lack et al., we calculate modulation transfer function MTF(f), noise power spectrum NPS(f) and detective quantum efficiency DQE(f) curves. The complete optical stack is modeled, including scintillator, fiber optic plate (FOP), optical adhesive and CMOS image sensor. For critical parts of the stack, detailed models have been developed, taking into account their respective microstructure. This includes two different scintillator types: Gd2O2S:Tb (GOS) and CsI:Tl. The granular structure of the former is modeled using anisotropic Mie scattering. The columnar structure of the latter is introduced into calculations directly, using the parameterization capabilities of GEANT4. The underlying homogeneous CsI layer is also incorporated into the model as well as the optional reflective layer on top of the scintillator screen or the protective polymer top coat. The FOP is modeled as an array of hexagonal bundles of fibers. The simulated CMOS stack consists of layers of Si3N4 and SiO2 on top of a silicon pixel array. The model is validated against measurements of various test detector structures, using different x-ray spectra (RQA5 and RQA-M2), showing good match between calculated and measured MTF(f) and DQE(f) curves.