The differential phase Q value method is adopted to investigate carried-suppressed return-to-zero (CSRZ), return-to-zero differential phase shift keying (RZ-DPSK), and optical duobinary (ODB) codes in 32 × 40 Gb/s DWDM system. The relationship between the transmission distance and optical output power is compared in order to select the optimal code. Furthermore, mixed phase noise of the spontaneous emission noise (ASE), Gordon-Mollenauer (GM) noise, and intrachannel four wave mixing (IFWM) effect has been analyzed. The numerical results show that DPSK is the most suitable for the DWDM system and adjustable range of output optical power is up to 10.2 dBm at 1200 km. Besides, ASE noise is dominant in degradation of received signal quality. However, RZ-DPSK and CSRZ show better performance for suppression of ASE noise, which can serve as a guideline for upgrading in current state-of-art deployed system.