This paper compares the transmission properties of different modulation formats in 10Gb/s system through computer simulation. The modulation formats include the following four formats, they are return to zero (RZ), non-return to zero (NRZ), carrier-suppressed return-to-zero (CS-RZ) and optical duobinary. We compare these four formats in terms of the capability of combating nonlinear effects such as self-phase modulation (SPM), cross-phase modulation (XPM). And we also compare the chromatic dispersion tolerance of these four formats. It is shown that optical duobinary has wider chromatic dispersion tolerance relative to other formats. And CS-RZ has better effect on anti-nonlinearity.