River segmentation is an important aspect of remote sensing image analysis. In the actual remote sensing images of rivers, the background is mostly complex and heterogeneous, the traditional detection methods cannot identify the small tributaries and the edge information is rough. To solve the aforementioned problems, we propose a separable attention network based on different size fusion. In this method, residual neural network is used as the backbone network to obtain the information features of rivers, and the deep feature information is fused with the shallow feature information through attention modules of different scales. The shallow feature and large-scale attention module are used to locate the main position of the river, while the deep feature and small-scale attention module are responsible for the fine segmentation of the river edge, so as to accurately extract the river from the background. The experimental results show that, compared with previous methods, the accuracy and detection speed of proposed method are significantly improved. The mean intersection over union (MIoU) on the river segmentation data set reaches 97.07%. Meanwhile, this model has excellent generalization performance, and the MIoU of the public data set CamVid reaches 69.4%.