Rotary inertial navigation modulation mechanism can greatly improve the inertial navigation system (INS) accuracy through the rotation. Based on the single-axis rotational inertial navigation system (RINS), a self-calibration method is put forward. The whole system is applied with the rotation modulation technique so that whole inertial measurement unit (IMU) of system can rotate around the motor shaft without any external input. In the process of modulation, some important errors can be decoupled. Coupled with the initial position information and attitude information of the system as the reference, the velocity errors and attitude errors in the rotation are used as measurement to perform Kalman filtering to estimate part of important errors of the system after which the errors can be compensated into the system. The simulation results show that the method can complete the self-calibration of the single-axis RINS in 15 minutes and estimate gyro drifts of three-axis, the installation error angle of the IMU and the scale factor error of the gyro on z-axis. The calibration accuracy of optic gyro drifts could be about 0.003°/h (1σ) as well as the scale factor error could be about 1 parts per million (1σ). The errors estimate reaches the system requirements which can effectively improve the longtime navigation accuracy of the vehicle or the boat.