Spectral variability is an inevitable problem in spectral unmixing. The linear mixing model (LMM) is often used due to its simplicity and mathematical tractability. Unfortunately, the linear mixing assumption is not always true in many real scenarios. To address this issue, we adopt a variant of the LMM to take care of spectral variability, called scaled and perturbed LMM, which can be used to constrain modeling reflectance scaling caused by topography or illumination, and simulating irregular spectral variabilities. To facilitate effective optimizations of the variables, a few regularizations are employed to regularize the introduced constraints and an alternating direction method of multipliers algorithm is further used to optimize all the variables of this model. Experimental results obtained from a synthetic dataset and two real datasets demonstrate that the proposed approach outperforms other algorithms in unmixing hyperspectral images with spectral variabilities.