Today and next generation optical coherent systems rely more and more in DSP algorithms to improve capacity, spectral
efficiency and fiber impairments mitigation. The amount of signal processing is remarkable, and because of that ASICs
are preferable in order to comply with cost, power consumption and size, required in OIF 100G optical module
standards. One important step in the ASIC development process is the validation of the DSP algorithms mathematical
models in a high level language that consider HW characteristics and constrains. In this work we present, compare and
evaluate in experimental data the mathematical model developed in Matlab and the SystemC model developed in C++.
The DSP algorithms functionalities implemented were orthonormalization, CD equalizer, clock recovery, dynamic
equalizer, frequency offset and phase estimation. The SystemC model considers clock signals, reset/enable structures,
parallelization, finite fixed-point operations and structures that are closer to the ASIC HW implementation; due to these
restrictions the performance is not as good as the mathematical modeling. The DSP algorithms models are evaluated in
two 112 Gbit/s DP-QPSK experimental scenarios. In the first scenario the models are evaluated in back-to-back with
ASE noise loading; in the second scenario the models are compared in a 226km optical fiber recirculation loop, with
80x112 Gbit/s DP-QPSK channels (8.96 Tbit/s). In the back-to-back experiment the OSNR penalty from the
mathematical model to the SystemC model is only 1,0dB and in the recirculation loop the maximum reach is 2,600 km
and 2,200 km for the Matlab and SystemC models respectively.