Dual-frequency heterodyne interferometer has been widely applied in many fields. It is also an important part for measuring position error in the system that laser simultaneously measure six degree of freedom geometric motion errors of a linear guide. To segregate the heat source, simplify the optical path and obtain a convenient delivery, a polarization maintain fiber (PMF) is used in the system for linking the laser source and the measurement section. However, the nonlinear error caused by PMF will reduce the measurement accuracy, and cannot be ignored in ultra-precision measurement. In this paper, a measurement model for analyzing the nonlinear error caused by the axis misalignment between the laser source and PMF was established with the Jones Matrix. Experiments were performed for verifying and calculating. The results show that the measuring error caused by nonlinear error is less than 2 nm with the axis misalignment being up to 10 degrees.