The field of image restoration lacks a promising comparison vehicle for judging the effectiveness of competing algorithms. By far the most widely adopted quantitative measurement of image restoration performance is by means of the SNR improvement. We address the issues of performance assessment of image restoration and propose a unified framework for the performance measure. The SNR improvement, which is within the proposed framework, is shown to be an inappropriate performance measure for image restoration. By introducing a metric for pixel fidelity improvement and incorporating the main properties of the human visual system into the measurement, we devise a performance measure of better quality, particularly of higher precision.