The nuclear magnetic resonance gyroscope is based on spin-exchange optical pumping of noble gases to detect and measure the angular velocity of the carrier, but it would be challenging to measure the precession signal of noble gas nuclei directly. To solve the problem, the primary detection method utilizes alkali atoms, the precession of nuclear magnetization modulates the alkali atoms at the Larmor frequency of nuclei, relatively speaking, and it is easier to detect the precession signal of alkali atoms. The precession frequency of alkali atoms is detected by the rotation angle of linearly polarized probe light; and differential detection method is commonly used in NMRG in order to detect the linearly polarized light rotation angle. Thus, the detection accuracy of differential detection system will affect the sensitivity of the NMRG. For the purpose of further improvement of the sensitivity level of the NMRG, this paper focuses on the aspects of signal detection, and aims to do an error analysis as well as an experimental research of the linearly light rotation angle detection. Through the theoretical analysis and the experimental illustration, we found that the extinction ratio σ2 and DC bias are the factors that will produce detective noise in the differential detection method.