The prime limitation of optical sensors is the need for external sources of illumination while capturing the scene. This prevents them from recognizing objects in extreme conditions, such as insufficient illumination or severe weather (e.g., under fog or smoke). The thermal imaging sensors have been introduced to circumvent this deficiency, which acquires the image based on thermal radiation emitted by the objects. The technological advancement in thermal imaging enables the visualization of objects beyond the visible range that promotes its use in many principal applications, such as military, medical, agriculture, etc. However, hardware point of view, the cost of a thermal camera is prohibitively higher than that of an equivalent optical sensor. This led to employ software-driven approaches called super-resolution (SR) to enhance the resolution of given thermal images. We propose a deep neural network architecture referred to as “ThermISRnet” as the extension of our earlier winner architecture in the Perception Beyond the Visible Spectrum (PBVS) thermal SR challenge. We use a progressive upscaling strategy with asymmetrical residual learning in the network, which is computationally efficient for different upscaling factors such as ×2, ×3, and ×4. The proposed architecture consists of different modules for low- and high-frequency feature extraction along with upsampling blocks. The effectiveness of the proposed architecture in ThermISRnet is verified by evaluating it with different datasets. The obtained results indicate superior performance as compared to other state-of-the-art SR methods.