The mode transition of high-order cladding modes in a coated long-period fiber grating (LPFG) was studied using coupled-mode theory, the response characteristic of cladding mode effective index was analyzed with increasing overlay thickness. It was found through analyses that the shift of resonant wavelength in the mode transition region was larger than that in the non-mode transition region. Further more, the phase-matching curves for 19th cladding mode was investigated when the overlay thickness was located in the mode transition and non-mode transition regions, the shift between two resonance wavelengths of dual peak in the mode transition region was bigger than that in non-transition mode region. The response characteristics of film refractive index of coated LPFG was investigated for a high-order cladding mode while the overlay thickness was located in mode transition and non-transition mode regions, the positions and amplitudes of the dual peak of the high-order cladding mode had different changes when the film refractive index vary slightly. The sensor sensitivity was defined and calculated. The result shows that the sensitivity of film refractive index could reach 69.12 and 0 respectively, when the overlay thickness was located in mode transition and non-mode transition regions. The corresponding resolutions of film refractive index can be available to 10-7 in mode transition region.