In this paper, we introduce circle 16 quadrature amplitude modulation (C-16QAM) modulation format in a high spectral efficiency zero-guard-interval (ZGI) coherent optical (CO) orthogonal frequency-division multiplexing (OFDM) system. At transmitter, the C-16QAM has advantages over the conventional square 16QAM in terms of transmission distance and tolerance to laser linewidth and fiber nonlinearities. ZGI CO-OFDM enables to take away the cyclic prefix (CP), so it has the benefit of higher spectral efficiency compared with the conventional CO-OFDM system. At receiver, in order to compensate chromatic dispersion (CD) and phase noise in a single channel ZGI CO-OFDM system, we studied the overlapped frequency domain equalizer (OFDE) and two carrier phase recovery (CPR) algorithms. We simulate the above systems and the results demonstrate that with the C-16QAM, a 28GBaud ZGI CO-OFDM system can have the longer transmission distance, the higher tolerance to laser linewidth and fiber nonlinearities with contrast to the conventional square 16QAM.