It is a difficult issue in image restoration to eliminate noise while avoiding the staircase effect and preserving edges. The anisotropic diffusion model proposed by Perona and Malik (PM) and the total variation model presented by Rudin, Osher, and Fatemi (ROF) are widely used to restore an image. However, the well-known defect of the two classic models is that they tend to cause the staircase effect. We propose a well-balanced anisotropic diffusion (WBAD) model by considering an adaptive balance parameter. The balance can be made in a selective way, meaning that it will alternate between the PM diffusion and ROF diffusion in accordance with the image features. The proposed WBAD model can preserve edges well while reducing noise, but it also causes less staircasing effect in the less smooth regions because it acts like the PM diffusion in these regions. Considering that the fourth-order PDEs can reduce the staircasing effect, we introduce a hybrid image restoration model based on an adaptive weight parameter to take advantage of the WBAD model and the fourth-order model. The experimental results illustrate that our algorithm can effectively remove noise and preserve edges. The higher values of peak signal-to-noise ratio and MSSIM highlight the better performance of our hybrid image restoration model.