A non-uniform QAM OFDM modulation system based on probabilistic shaping (PS) and geometric shaping (GS) is proposed in this paper, which can effectively resist the interference among signal waveforms and improve spectrum utilization and channel capacity. First, Huffman coding is used in the scheme for probability shaping to reduce the average energy of the signal. Uniform 16QAM is adjusted to non-uniform 12QAM and uniform 32QAM is adjusted to non-uniform 12QAM. Then geometric shaping is applied. Taking generalized mutual information (GMI) as the objective function, pairwise optimization (PO) algorithm is applied to continuously adjust the constellation. Finally, the coordinate point with the largest mutual information is obtained. The simulation results show the performance of non-uniform 12QAM and non-uniform 24QAM under different conditions, we obtain the signal-to-noise ratio (SNR) with generalized mutual information curve, signal-to-noise ratio with bit error rate (BER) curve under different QAM modulation formats and the bit error rate with the different length of the fiber under different transmission modes. The results show that the proposed non-uniform 12QAM-GS-OFDM has a gain of 0.02 bit/symbol over 12QAM-OFDM, the proposed non-uniform 24QAM-GS-OFDM has a gain of 0.02 bit/symbol over 24QAM-OFDM.
An amplitude-phase precode method that based on a high speed ROF-MIMO system is proposed to solve the distortion problem due to the features of optical devices and link damages. This method uses the digital predistortion algorithm model，which is based on Weierstrass theorem. This could precompensation to the ROF links and reduce the influence of nonlinear characteristics of optical devices such as erbium-doped fiber amplifiers on system signal transmission, and solve the sensitivity of QAM, OFDM and other non-constant envelope modulation signals to system distortion. Then the Minimum Mean Square Error (MMSE) precoding method is used to optimize the spatial characteristics of the MIMO transmission signal to solve the signal interference problem, so as to reduce the bit error rate of the system and reduce the complexity of digital signal processing at the user terminal. The simulation results show that the bit error rate （BER） performance of ROF system with the proposed amplitude-phase precode method is lower than 4*10<sup>-4</sup> during the transmitting 10 Gb/s 16QAM-OFDM signal in 20km standard single model fiber. The proposed scheme has the obvious lower bit error rate than the non-precode high speed ROF-MIMO system. Therefore, the proposed scheme improves the base station and antenna array coverage, at same time, reduce the complexity of terminal unit and increase the stability of the system transmission.