1.

Introduction

An image captured by an imaging system represents the degraded version of an original image due to blurring and additive noise.^{1, 2} For a size $M\times N$ image, a typical degradation model can be written as

where the vectors $\mathbf{x}$ , $\mathbf{y}$ , and $\mathbf{n}$ are the lexicographically ordered original image, the observed image, and the additive noise of size $MN\times 1$ , respectively. In addition, $\mathbf{H}$ is the degradation matrix of size $MN\times MN$ to represent a spatially invariant or spatially varying point spread function (PSF).

Regularized constrained least squares (RCLS) have been widely used to obtain a solution for Eq. 1, and the solution is obtained by minimizing the following function with respect to $\mathbf{x}$ ,^{1, 2}

where $\alpha$ denotes the regularization parameter to control the tradeoff between fidelity to the data and smoothness, and $\mathbf{C}$ typically represents a high-pass operator.

The prior knowledge used in RCLS is that the original image is smooth. However, this is a global requirement and therefore not effective in terms of local smoothness. Also, when an iterative technique is used to obtain the solution of Eq. 2, the iterative solution may suffer from noise amplification after a certain iteration step when the additive noise is serious.^{2, 3, 4} Therefore, a more desirable solution can be obtained by imposing reasonable constraints into the solution space of RCLS.^{3, 4, 5} Projection onto convex sets (POCS) has been widely used in many areas, since it is effective to impose nonlinear local properties into the solution space.^{3, 6}

In this work, an iterative adaptive hybrid image restoration algorithm using a local smoothing constraint is presented. We follow the formulation represented by Eq. 2 and propose to bring knowledge about local properties of the original image into the restoration process, so that prior knowledge and spatial adaptivity are incorporated in the solution. The basic motivation is to constrain local ranges of values that the restored image can take, leading to increased convergence speed of the iterative algorithm and performance improvement compared to RCLS.

This work is organized as follows. In Sec. 2, the proposed hybrid gradient-projection restoration algorithm is explained. Experimental results and conclusions are presented in Sec. 3.

2.

Proposed Algorithm

The gradient iteration of the regularized smoothing functional in Eq. 2 can be written as

There exist various ways for determining the regularization parameter $\alpha$ .^{1, 2, 7} According to Ref. 7, we determine the regularization parameter bywhere $\theta ⩾2{\Vert \mathbf{y}\Vert }^2$ . The iterative solution in Eq. 3 reflects only global smoothing requirements and therefore it is not effective in terms of local smoothness. When a local smoothing constraint is imposed into the solution space of RCLS, the iteration in Eq. 3 can take the formwhere $P$ represents a projection operator of a signal onto a set with desirable local properties. The proposed projection set is defined by the local statistics of the partially restored image at each iteration step. For a pixel of the partially restored image ${\widehat{x}}_k(i,j)$ , the local mean and variance at coordinate $(i,j)$ with window $W$ are defined aswhere $W=(2U+1)\times (2V+1)$ is the extent of the analysis window that is symmetric about the point $(i,j)$ . In addition, the local maximum of the partially restored image at point $(i,j)$ is defined aswhere $S_{(i,j)}$ is the support region that determines the local maximum about the point $(i,j)$ . In this work, it is same with the analysis window used for the local mean and variance. Using the local information in Eqs. 6, 7, a parameter controlling the degree of local smoothing is defined asfor $L$ bits per pixel. In Eq. 8, $T_k$ is a threshold to be determined and $B(i,j)$ is defined asIn Eq. 9, $SW$ and $LW$ represent the relatively small and large window sizes centered at point $(i,j)$ .

In general, noise amplification contributes to the problem of restoration, and therefore desirable properties such as detection of noisy pixels and the degree of local activity should be incorporated into the restoration process to effectively suppress noise amplification. It is clear how the local statistics affect $B(i,j)$ . Let us assume that ${\widehat{x}}_k(i,j)$ is a noisy pixel belonging to an element of a homogeneous activity region. In such a case, ${\sigma }_{{\widehat{x}}_{k,SW}}^2(i,j)$ is greater than ${\sigma }_{{\widehat{x}}_{k,LW}}^2(i,j)$ , and therefore local smoothing is necessary to suppress the noise amplification. As the noise increases, the variance ratio decreases, leading to smaller $B(i,j)$ . In addition, when the same noise is added to low and high activity regions, the noise within the low activity region is more visible than that within the high activity region. Therefore, tighter bounds should be applied to the noisy pixel within the low activity region than to that within the high activity region. ${\widehat{x}}_{k,SW,\max }(i,j)∕m_{{\widehat{x}}_{k,SW}}(i,j)$ in Eq. 9 is used as a way to represent the degree of local activity in this work. As the local activity decreases, $B(i,j)$ decreases. A smaller $B(i,j)$ leads to tighter bounds, while looser bounds are for a larger $B(i,j)$ , so that the noise amplification is effectively suppressed. They are in agreement with the noise masking property in areas of high spatial activity of the human visual system.⁸

Local information has the limit to effectively suppress noise amplification. Therefore, global information about the noise can help to effectively reduce noise amplification. For example, tighter bounds are more desirable as the additive noise increases. $T_k$ in Eq. 8 is used to incorporate global information into the restoration process, so that it can control the degree of bounds that is computed at each iteration step, such as

where $Z$ represents a constant, and ${\mathbf{n}}_{k-1}=\mathbf{y}-\mathbf{H}{\mathbf{x}}_{k-1}$ .

The proposed algorithm is used to obtain a solution that is an element of the intersection set between a solution space using the gradient approach reflecting the global smoothing constraint, and a projection set incorporating the local smoothing constraint.

3.

Experimental Results and Conclusions

The proposed adaptive hybrid image restoration algorithm is tested with various noisy blurred images and degradation, and it is compared to typical RCLS. In the set of such experiments, the $256\times 256\text{pixels}$ Lena and Cameraman images are described here. The original images were blurred by $7\times 7$ motion blur and Gaussian blur with variance of 5. A Gaussian-distributed noise signal was added to the blurred images. We tested the proposed algorithm for various SNRs. In addition, a 2-D Laplacian operator was used for the high-pass operator in Eq. 2.² For evaluating the objective and subjective performance of the algorithm, the mean squared error (MSE) and structural similarity index (SSIM) in Ref. 9 were used, where SSIM takes a value between $-1$ (the worst value) and 1 (the best value). Also,

was used for terminating the iteration. In these experiments, $m=5$ was used for 5- and $10\text{-}\mathrm{dB}$ noises. Also, $m=6$ and $m=7$ were used for 20- and $30\text{-}\mathrm{dB}$ noises, respectively. In addition, $SW=3$ and $LW=7$ were used for small and large size windows in Eq. 9.

Figures 1, 1, 1 show the degraded Lena image ( $7\times 7$ Gaussian blur with variance of 5- and $10\text{-}\mathrm{dB}$ Gaussian noise), the restored image with RCLS, and the restored image with the proposed method, respectively. For $Z=0.07$ , the proposed algorithm converges after ten iterations ( $\mathrm{MSE}=260$ and $\mathrm{SSIM}=0.677$ ), while RCLS requires 39 iterations ( $\mathrm{MSE}=493$ and $\mathrm{SSIM}=0.432$ ). The results show that RCLS leads to noise amplification in the restored image. On the other hand, the proposed hybrid algorithm effectively suppresses the noise amplification. When tighter bounds (smaller $Z$ ) are used, the convergence becomes faster. However, tighter bounds result in oversmoothed images. On the basis of our experiments, $0.05⩽Z⩽0.1$ is a good range with convergence speed and performance.

Figures 2 and 2 show MSE and convergence speed comparisons as a function of iteration number for Lena image when $10\text{-}\mathrm{dB}$ Gaussian noise is added. The results show that the proposed algorithm has the capability to keep a minimum mean squared error, while RCLS is very sensitive to the terminating criterion, since the noise is amplified after a certain iteration number. Also, it is verified that the proposed algorithm is faster than RCLS for all cases.

Tables 1, 2 summarize the performance comparisons (MSE, SSIM, and iteration number) at convergence for Gaussian blur and motion blur, respectively. As expected, the performance gain of the proposed algorithm increases as the additive noise increases. However, as the additive noise decreases, looser bounds are necessary to avoid oversmoothed images. As shown in Tables 1, 2, the proposed algorithm performs similarly to RCLS for additive noise less than $30\mathrm{dB}$ .

Table 1

Performance comparisons at convergence (Lena).

Noise(dB)	Method	Gaussian blur			Motion blur
		MSE	SSIM	Iteration Number	MSE	SSIM	Iteration Number
5	RCLS	1799	0.230	66	2336	0.204	66
	Hybrid	303	0.612	10	288	0.633	9
10	RCLS	493	0.432	39	797	0.352	49
	Hybrid	260	0.677	10	245	0.703	9
20	RCLS	212	0.647	64	241	0.598	74
	Hybrid	189	0.730	38	174	0.734	53
30	RCLS	139	0.772	123	125	0.773	127
	Hybrid	139	0.772	123	125	0.773	127

Table 2

Performance comparisons at convergence (Cameraman).

Noise(dB)	Method	Gaussian blur			Motion blur
		MSE	SSIM	Iteration Number	MSE	SSIM	Iteration Number
5	RCLS	2423	0.161	64	3071	0.150	63
	Hybrid	471	0.495	9	458	0.506	8
10	RCLS	691	0.304	38	1049	0.252	47
	Hybrid	409	0.592	9	393	0.605	8
20	RCLS	324	0.490	66	350	0.446	74
	Hybrid	305	0.642	35	280	0.636	54
30	RCLS	202	0.677	135	182	0.671	134
	Hybrid	201	0.678	133	181	0.671	134

The novelty of the proposed algorithm is that it leads to faster convergence speed, and subjectively and objectively better performance than typical RCLS without prior information of an original image by incorporating the local smoothing constraint.

We propose an iterative adaptive hybrid image restoration algorithm using a local smoothing constraint. Each pixel in an image is projected onto a local smoothing set determined by the local mean, variance, and maximum intensity value of the partially restored image. The ratio of these parameters is used to define the convex set, resulting in effective suppression of the noise amplification and faster convergence speed.