In this paper, the effects of gridless spectrum allocation in Wavelength Division Multiplexed (WDM) optical networks are examined. The advanced modulation formats and multi-rate transmissions of the signals, which are key parameters in the optical system project, are taken into account. The consumed spectrum, as well as the impact of linear and nonlinear impairments on the signal transmission, are compared to WDM network adopting standard grid and gridless ITU. To analyze the influence of these physical effects, some key network design parameters are monitored and evaluated, such as the guard band size, the signal occupied bandwidth, the laser power and the quality of channels. The applied signal modulation formats were On/Off Keying (OOK), Quadrature Phase Shift keying (QPSK), and Dual Polarization State Phase Modulation (DP-QPSK), whereas the transmission rate per wavelength was varied from 10 Gb/s to 100Ghz. The guard band, signal band, and laser power were swept and the resulted Bit Error Rate (BER) was estimated from the eye-diagram. Analytical calculations and simulations are conducted in order to evaluate the impact of the gridless spectrum allocation on both the spectral consumption and the signal quality of transmission (QoT). Results reveal that a gridless transmission system reduces the spectral consumption while offering an acceptable QoT. This work was carried out with both analytical modeling and numerical calculation using the Optisystem as well as Matlab.