Hyperspectral image (HSI) restoration is a technique to inverse the information degradation process that occurs on a hyperspectral imaging system, i.e., spectrometer. Spectrometers can be classified as two types: plane-scanning and line-scanning spectrometers. It is necessary for a restoration algorithm to match the corresponding degradation process. However, most current restoration algorithms are only suitable to the former one. To solve such a mismatch of restoration algorithms to the imaging process in this paper, a new framework of HSI restoration is proposed. Compared to the existing frameworks, the proposed one is more applicable to a line-scanning spectrometer. Moreover, to solve the ill-posedness of such a framework, an anisotropy regularization term combining a vertical total variation and a linear spectral mixture is designed. Experimental results based on two simulation datasets, Pavia and San Diego, proved the effectiveness of the proposed framework and regularization term.