Single image super-resolution (SISR), which aims at obtaining a high-resolution image from a single low-resolution image, is a classical problem in computer vision. In this paper, we address this problem based on a deep learning method with residual learning in an end-to-end manner. We propose a novel residual-network architecture, Residual networks of Residual networks (RoR), to promote the learning capability of residual networks for SISR. In residual network, the signal can be directly propagated from one unit to any other units in both forward and backward passes when using identity mapping as the skip connections. Based on it, we add level-wise connections upon original residual networks, to dig the optimization ability of residual networks. Our experiments demonstrate the effectiveness and versatility of RoR, it can get a faster convergence speed and gain higher resolution accuracy from considerably increased depth.