In this paper we investigate limits of intensity and phase modulation formats used in optical communications. Non- Return to Zero, Return to Zero, Chirped Return to Zero, Carrier-Suppressed Return to Zero, Binary Phase Shift Keying, and Quadrature Phase Shift Keying including the most actual solutions, such as Polarization Division Multiplexing Quadrature Phase-Shift Keying, are investigated in terms of spectral efficiency, Bit Error Rate to find the limits for selected topologies and spectral grids in Dense Wavelength Division Multiplexing. Differential Phase-Shift Keying and mainly Differential Quadrature Phase-Shift Keying offer improvements in Bit Error Rate and transmission reach, among others. There are practical conclusions about transition from 10 Gb•s-1 to much higher bit rates. We study the potential increase of efficiency of Wavelength Division Multiplexing. We investigate the performance of Polarization Division Multiplexing Quadrature Phase-Shift Keying in very high speed optical systems that are promising even for terabit transmission.