Deep learning has enabled substantial progress in crowd counting, but existing methods still face difficulties due to significant scale variations, severe occlusion, and overcrowding. To explicitly address these problems, we propose a crowd counting method based on the inverse attention residual network. Initially, the first 10 layers of VGG-16 with a strong transfer learning ability and a feature extraction ability are used as the front-end network for preliminary extraction of head features. Afterward, we use the inverse attention residual module to divide the background information and crowd information and retain the accurate pixel position information to reduce the impact of various types of noise in the input image. Finally, the mini multiscale dilated convolution is used as the backbone of the back-end network to perform the aggregation of feature maps, so the network can learn the details of heads at different scales and use the 1  ×  1 convolutional layer to regress the feature maps to the density map. As a result, the above problems are resolved to a certain extent. Experimental results show that the mean absolute error and the mean square error of the proposed algorithm are significantly lower than those of the comparison algorithms.