Adaptive optics (AO) systems deliver high-resolution images that may be ideal for precisely measuring positions of stars (i.e., astrometry) if the system has stable and well-calibrated geometric optical distortions. A calibration unit equipped with a back-illuminated pinhole mask can be utilized to measure instrumental optical distortions. AO systems on the largest ground-based telescopes, such as the W. M. Keck Observatory and the Thirty Meter Telescope (TMT), require pinhole positions known to be ∼20  nm to achieve an astrometric precision of 0.001 of a resolution element. In pursuit of that goal, we characterize a photolithographic pinhole mask and explore the systematic errors that result from different experimental setups. We characterized the nonlinear geometric distortion of a simple imaging system using the mask, and we measured 857-nm root mean square of optical distortion with a final residual of 39 nm (equivalent to 20  μ for TMT). We use a sixth-order bivariate Legendre polynomial to model the optical distortion and allow the reference positions of the individual pinholes to vary. The nonlinear deviations in the pinhole pattern with respect to the manufacturing design of a square pattern are 47.2 nm ± 4.5 nm (random) ± 10.8 nm (systematic) over an area of 1788  mm². These deviations reflect the additional error induced when assuming that the pinhole mask is manufactured perfectly square. We also find that ordered mask distortions are significantly more difficult to characterize than random mask distortions as the ordered distortions can alias into optical camera distortion. Future design simulations for astrometric calibration units should include ordered mask distortions. We conclude that photolithographic pinhole masks are >10 times better than the pinhole masks deployed in first-generation AO systems and are sufficient to meet the distortion calibration requirements for the upcoming 30-m-class telescopes.