In recent years, ecological and environmental degradations have occurred widely around the world, especially in urban areas with rapid economic development, intensive human activities, and massive urbanization. We aim at proposing an effective and comprehensive urban ecological environment evaluation method based on remote sensing. Harmonic analysis of time series method and spatial–temporal information fusion based on nonlocal means filter was used to improve the quality of long-term remote sensing data from 2003 to 2019. The greenness, dryness, heat, and atmospheric turbidity of the ecological environment were represented by normalized difference vegetation index, normalized difference built-up index, land surface temperature, and aerosol optical depth, respectively. An urban ecological comfort index (UECI) was established by the weighted combination of the above four indicators using entropy method. UECI was calculated at seasonal and annual scales from 2003 to 2019 in Hefei city, China, as a case study. Furthermore, Mann–Kendall trend test, spatial autocorrelation analysis, and spatial clustering analysis were performed on the UECI results. The experimental results show that UECI in Hefei had an increasing trend from 0.5446 to 0.5737 between 2003 and 2019, indicating that its ecological environment is actively improving. UECI is roughly equal in spring, summer, and autumn, and the worst is in winter. The responses of UECI to urban expansion, environmental protection policies, and extreme weather were also explored. Its sensitively analysis results and the fact that the spatial–temporal trends of UECI were consistent with the influencing factors reflected the stability and reliability of UECI, indicating that the proposed UECI performed well in Hefei and could be applied to measure urban ecological comfort level in other study areas.