The stereo light microscope (SLM) plays an important role in the measurement of three-dimensional geometry on the microscopic scale. We propose a fast and precise affine calibration algorithm based on the invariable extrinsic parameters for the SLM. This calibration algorithm with a free planar reference consists of three steps: first, derive the extrinsic parameters based on their invariable definition in the pinhole and affine models; second, calculate the intrinsic parameters through homography matrix; finally, refine all the model parameters by global optimization with the previous closed-form solutions as the initial values. The effectiveness of assuming a noncoaxial optical system as an affine camera is also verified to affinely model all types of SLMs. The calibration experiments show that the affine calibration is preferable for multicriteria including running time, relative positioning precision, and absolute positioning precision. With PlanApo S 1.5× and a total magnification of 3.024× , the proposed affine calibration algorithm achieves a distance error of 0.423 μm and a positioning error of 0.195 mm within 10.6 s.