To meet the increasing bandwidth demand for rapidly growing data flow from the wide spreading multimedia services, broadband Internet, and cloud services, 100G and even beyond data transmission based on compact and low-cost transceivers have been intensively studied in short-reach links, access, inter data-center-interconnections, metro and regional networks. For these systems, direct-detection is more attractive due to the lower system cost and power consumption. In this paper, we review four advanced modulation formats including pulse amplitude modulation-4 (PAM-4), carrier-less amplitude and phase modulation-16 QAM (CAP-16), discrete Fourier transform spread orthogonal frequency division multiplexing (DFT-S OFDM) and Duobinary with low-cost direct detection in 100 Gb/s/λ short-reach and metro optical transmission systems. These modulation formats are all experimentally demonstrated with similar digital signal processing algorithms. Three kinds of chromatic dispersion (CD) compensation ways named pre-CD method, single sideband (SSB) and dispersion compensating fiber (DCF) are also compared. Two types of modulators including IQ modulator and dual-drive Mach-Zehnder modulator (DDMZM) are employed to generate SSB or pre-CD signal in this experiment and their transmission performance is also evaluated. To further to improve the spectrum efficiency and reduce the cost, we also present a detailed comparison of applying three advanced modulation formats including pulse amplitude modulation-8 (PAM-8), carrier-less amplitude and phase modulation-64 QAM (CAP-64), and discrete Fourier transform spread orthogonal frequency division multiplexing (DFT-S OFDM) with a bandwidth-limited direct-detection receiver for 100 Gb/s/λ optical transmission systems. The comparison is carried out to evaluate the performance of each modulation format in terms of nonlinear equalization, received optical power and optical signal to noise ratio (OSNR).