With the increase of fiber laser output power, stimulated Raman scattering (SRS) and transverse mode instability (TMI) effects have become the main factors limiting the power boost of high-power, high-brightness fiber lasers. In this work, the TMI effect in different core diameter fiber laser oscillators are studied experimentally. In the 25/400YDF laser oscillator, the TMI threshold is around 1780W, while the TMI threshold of the 30/400YDF is around 1070W. A new type gain fiber (25-30-25YDF) by precisely splicing 25/400YDF with 30/400YDF is employed in the fiber oscillator. When the launched pump power is ~2044W, the maximum average power of the 25-30-25YDF laser oscillator is up to ~1573W without any sign of the TMI. The experimental results show that the larger the efficient core diameter of the gain fiber, the lower TMI threshold of the fiber laser oscillator under the identical experimental structure. By optimizing the length of different core diameter fibers in the cavity and improving the fusion quality of each splice point of the laser, the output average power and beam quality of the fiber laser can be further improved.