In this paper, nonlinear noises that characterize the performance of a long-haul optical fiber sensing system were investigated. In a 50 km transmission system, when stimulated Brillouin scattering (SBS) occurs seriously, the phase noise of the interferometer increases from -102dB (0dB=1rad/sqrt(Hz)) to -84dB due to the enlargement of the laser linewidth and the deterioration of the signal-to-noise ratio (SNR). While the phase modulation (PM) and the Phase-generated carrier (PGC) modulation to the laser frequency are applied simultaneously, the suppression of SBS is 35dB and 10dB respectively in the backscattering spectra and the interferometric phase noise caused by SBS is completely eliminated. When the input power continues to increase and exceeds the modulation instability (MI) threshold, the system performance also deteriorates significantly. The forward output spectra of the 50 km optical fiber and phase noise of the interferometer are measured. The results show that with the increase of the injection power, the increase trend of the MI component in the total power of the spectrum is approximately consistent with that of the phase noise. It can be concluded that the phase noise introduced by MI is mainly caused by the increase of light intensity noise and the deterioration of optical SNR. Therefore, in order to reduce the impact of MI in the sensor system, it is needed to avoid the generation of serious MI as far as possible, and then the ultra-narrow band filter should be used to filter the MI sideband for the improvement of the system SNR.