To support the ever-increasing demand for high-speed optical communications, Nyquist spectral shaping serves as a promising technique to improve spectral efficiency (SE) by generating near-rectangular spectra with negligible crosstalk and inter-symbol interference in wavelength-division-multiplexed (WDM) systems. Compared with specially-designed optical methods, DSP-based electrical filters are more flexible as they can generate different filter shapes and modulation formats. However, such transmitter-side pre-filtering approach is sensitive to the limited taps of finite-impulse-response (FIR) filter, for the complexity of the required DSP and digital-to-analog converter (DAC) is limited by the cost and power consumption of optical transponder. In this paper, we investigate the performance and complexity of transmitter-side FIR-based DSP with polarization-division-multiplexing (PDM) high-order quadrature-amplitude-modulation (QAM) formats. Our results show that Nyquist 64-QAM, 16-QAM and QPSK WDM signals can be sufficiently generated by digital FIR filters with 57, 37, and 17 taps respectively. Then we explore the effects of the required spectral pre-emphasis, bandwidth and resolution on the performance of Nyquist-WDM systems. To obtain negligible OSNR penalty with a roll-off factor of 0.1, two-channel-interleaved DAC requires a Gaussian electrical filter with the bandwidth of 0.4–0.6 times of the symbol rate for PDM-64QAM, 0.35–0.65 times for PDM-16QAM, and 0.3–0.8 times for PDM-QPSK, with required DAC resolutions as 8, 7, 6 bits correspondingly. As a tradeoff, PDM-64QAM can be a promising candidate for SE improvement in next-generation optical metro networks.