Overfitting of artificial-neural-network-based nonlinear equalizer for multilevel signals in optical communication systems

Kai Ikuta; Yuta Otsuka; Moriya Nakamura

doi:10.1117/12.2545803

24 February 2020 Overfitting of artificial-neural-network-based nonlinear equalizer for multilevel signals in optical communication systems

Kai Ikuta, Yuta Otsuka, Moriya Nakamura

Author Affiliations +

Proceedings Volume 11299, AI and Optical Data Sciences; 1129916 (2020) https://doi.org/10.1117/12.2545803
Event: SPIE OPTO, 2020, San Francisco, California, United States

Abstract

We investigated the problem of overfitting of artificial neural networks (ANNs) which are used for digital nonlinear equalizers in optical communication systems. ANN-based digital signal processing (DSP) can be used in the time domain or the frequency domain to compensate for the optical nonlinearity. We have reported the advantages of ANN over Volterra series transfer function (VSTF) in terms of the computational complexity. However, when pseudo-random binary sequence (PRBS) data is used to evaluate the ANN-based nonlinear equalizers, the ANNs can potentially learn the repeated PRBS patterns, resulting in overestimation of the equalization performance. In this paper, we clarify that the overfitting of the ANNs hardly occurs or requires much more neuron units than used for common nonlinear equalization, if we employ multi-level modulation signals including simple 4-ary pulse-amplitude modulation (PAM4). In our study, we compared binary signal and 4-level signal generated by PRBS data. White Gaussian noise (WGN) was added to the signals. The ANN-based nonlinear equalizers were trained using a least mean squares (LMS) algorithm, to attempt to “compensate” for the noise. The signal quality was evaluated using the error vector magnitude (EVM). When the overfitting occurs, the EVM is improved by the nonlinear equalization. The results revealed that, in the case of 4-level signal, the influence of the overfitting was suppressed in comparison with the case of the binary signal.

Citation Download Citation

Kai Ikuta, Yuta Otsuka, and Moriya Nakamura "Overfitting of artificial-neural-network-based nonlinear equalizer for multilevel signals in optical communication systems", Proc. SPIE 11299, AI and Optical Data Sciences, 1129916 (24 February 2020); https://doi.org/10.1117/12.2545803

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
5 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Binary data

Nonlinear optics

Complex systems

Telecommunications

Optical communications

Modulation

Signal generators

Show All Keywords

Keywords/Phrases

Search In:

Publication Years