Two approaches are presented for detailed analysis of transverse mode instability in fiber amplifiers based on a scalar finite element beam propagation method (BPM). The first employs two beams: one propagating at a fundamental frequency and one de-tuned to the middle of the stimulated thermal Rayleigh scattering (STRS) gain peak. This method was found to suffer from a computational artifact causing it to converge in some cases to an unphysical solution. The second was based on the steady periodic method. This required more computational resources but was found to be reliable and not susceptible to the artifact mentioned above. This method was used to simulate step-index fiber amplifiers, large pitch photonic crystal fiber amplifiers, and a hybrid large pitch photonic bandgap fiber amplifier with reduced symmetry. Results for reference step index fiber amplifiers were found to be consistent with those obtained by other methods. The simulated instability threshold values all fell between 200 and 310 Watts showing relatively little variation among designs. Some areas for improvement in the method are discussed.