Fiber nonlinear (NLI) noise, as well as amplified spontaneous emission (ASE) noise, degrades the performance of coherent optical fiber communications. We find that the temporal distribution of NLI noise, which is unlike that of ASE noise and depends on the waveform of signal pulse, has an impact on the estimation of error vector magnitude (EVM) and bit error ratio (BER). The influences of NLI noise, as well as ASE noise, on EVM and BER are analyzed for quadrature phase-shift keying and M-ary quadrature amplitude modulation systems. Modified theoretical formulas are proposed to estimate EVM and BER for the system under the effect of NLI noise, ASE noise, or mixing of these two noises. For system with mixing of ASE and NLI noises, according to our model, the optimal channel power for the best EVM is not corresponding to the best optical-signal-to-noise ratio. Our model is validated by numerical simulations.