Transmitter in-phase/quadrature (IQ) mismatch is difficult to suppress by using traditional methods in coherent optical orthogonal frequency division multiplexing (CO-OFDM) system with higher data rate, as a result an effective compensation method to solve the problem is urgently needed. In this paper, a novel training symbol structure to compensate IQ mismatch is proposed. Based on this structure we deduce the compensation method in frequency domain, and by utilizing the compensation method we can achieve independent compensation of IQ mismatch as well as channel distortion and promote the system performance significantly with the data rate increasing by 30Gb/s compared with the traditional methods. Numerical simulation demonstrates that the proposed method has better BER performance compared with the traditional method at the data rate of 50Gb/s after 640km transmission. When the phase and amplitude mismatch are 15° and 3dB respectively, the OSNR penalty of proposed method can decrease by 6dB while BER reaches to 1×10<sup>-3</sup>. What’s more, the tolerance of amplitude mismatch and phase mismatch can be improved about 3.5dB and 20° by the proposed method, respectively.