As an important mission of communication reconnaissance field, the action recognition of radio signals has attracted great attention of researchers, where feature extraction of signals is considered as the most essential element. However, feature extraction is still a challenging task in action recognition owing to the remarkable distortions and rapid changes of signals. In this paper, a novel deep learning method is proposed for simultaneous feature extraction and action recognition of signals, which can automatically learn discriminative and robust features layer by layer in a supervised manner. Firstly, signals are sliced and stacked into a real-valued matrix, and then fed into a multi-scale deep convolution neural network. Unlike the same size kernels employed in traditional convolution networks, multi-scale convolution kernels are advanced to explore diverse spatial correlation of signal matrix. In each layer, multiple one-dimensional (1-D) convolution kernels are designed to filter out representative features. Then 1-D pooling operators are constructed to reduce the dimensionality of features to formulate the multi-scale feature maps. Finally a softmax classifier is used to recognize the types of actions. By learning a group of 1-D filters from a large number of labeled signals, our proposed deep convolutional network can automatically extract the fine signatures of signals and explore the latent difference between communication signals and non-communication signals. Moreover, the 1-D convolution is of low-complexity computation and can simplify the computation during the backpropagation of the neural network. Experimental results on real signals show that our proposed method is effective in recognizing communication signals and noncommunication signals.