In this work we present the theoretical study of transverse mode instability (TMI) in ytterbium (Yb)-doped rectangular core fibers with different core aspect ratios using the fast Fourier transform (FFT) beam propagation method (BPM). As expected, the rectangular core fiber with larger aspect ratio (AR.) offers more efficient heat dissipation than a circular core fiber. However, it is found that the rectangular core fiber does not benefit from the better heat dissipation to suppress the TMI when compared to the circular core counterpart. The temperature building in the rectangular core fiber decreases by up to 24.6% with a 10:1 aspect ratio core, while threshold pump power drops by up to 38.3% when compared with a circular core fiber with the same core area. Our study reveals that a smaller effective refractive index difference between modes and a weaker gain saturation effect compensate the thermal advantage from more efficient heat dissipation.