Robust aerial infrared target recognition with multi-scale and multi-angle characteristics is a key technique in infrared systems. However, traditional algorithms often fail to achieve a high accuracy and robustness due to simple features and classifiers. Moreover, deep learning algorithms mainly focus on improving accuracy with the price of high complexity. To address above issues, we propose a two-stage lightweight aerial infrared target recognition based on convolutional neural networks(CNN). We propose the coarse region extraction based on the local contrast in the first stage, which combines infrared image characteristics properly. In the second stage, we propose the find target recognition, which constructs lightweight CNN by changing network layers and convolution kernels. Experimental results demonstrate the algorithm proposed can achieve recognition for six kinds of aerial infrared target. Compared with other algorithms, our algorithm obtains higher accuracy and robustness.