1 April 2010 Real-time image restoration for digital multifocusing in a multiple color-filter aperture camera
Author Affiliations +
Optical Engineering, 49(4), 040502 (2010). doi:10.1117/1.3393885
A multiple color-filter aperture (MCA) can provide a single camera with depth information and multifocusing. However, the original version of the MCA system exhibits inherent limitations such as manual, empirical tuning parameters for the color channel registration and fusion (CRF) process. Furthermore, a CRF output image still contains undesired out-of-focus blur because of the finite-sized apertures and the lateral displacement of each color-filter aperture, which results in low exposure, color mixing, deviation of color convergence, and divergence of light rays. For overcoming these problems, we present a real-time image processing solution for digital multifocusing in a MCA system.
Kim, Lee, Maik, and Paik: Real-time image restoration for digital multifocusing in a multiple color-filter aperture camera



Computational cameras with multifocusing functions have combined optics and signal processing techniques to overcome limitations in a pure digital signal processing approach, such as in Ref. 1. The multiple color-filter aperture (MCA)2, 3 system is one such computational camera technique, by which a single camera system can estimate depth information of objects at different distances and perform multifocusing. The existing MCA system, however, exhibits several problems, including low exposure, color mixing, deviation of color convergence, and divergence of light rays. These problems result in nonideal point-spread functions (PSFs), which deteriorate the overall quality of the image even after color channel registration and fusion (CRF). In this work, we present: 1. fundamental principles of the MCA computational camera, 2. the automatic CRF process, and 3. a real-time image restoration system for improving the quality of images acquired by the MCA system. For the image restoration process, we first estimate the PSF based on the method proposed in Ref. 4, and then design the truncated constrained least-squares (TCLS) restoration filter using the estimated PSF. Most image restoration filters tend to amplify noise and produce ringing artifacts, and the proposed TCLS filter is not an exception. For removing such artifacts, we additionally propose a spatially adaptive noise smoothing (SANS) algorithm using adaptively weighted mixing between the color-channel-aligned image and the restored image by the TCLS filter.


Automatic Color Channel Registration and Fusion for the Multiple Color-Filter Aperture Camera

The aperture of an optical system is the opening device that adjusts the amount of incoming light. Generally, the center of an aperture is aligned to the optical axis of the lens, and the convergence pattern on the imaging plane forms either a point or a circular region. However, if the center of the aperture is not aligned to the optical axis, the image formation of the sensor deviates from the optical axis. As shown Fig. 1, the MCA system consists of three apertures with R, G, and B color filters. The main advantage of the MCA system is that it provides additional depth information of objects at different distances, as shown in Figs. 1, 1, 1. At the cost of additional depth information, the image acquired by the MCA system contains a certain amount of out-of-focus blur, which results from color misalignment. For solving the color misalignment problem, manual registration of color channels and fusion methods have been proposed in Refs. 2, 3.

Fig. 1

(a) The MCA camera system, (b) an object is on the in-focus position b, (c) an object is on the near-focus position c, and (d) an object is on the far-focus position a.


We propose cluster-based image segmentation and phase correlation matching (PCM) for automatically registering pixels in three color channels. The proposed cluster-based segmentation algorithm employs K-means clustering and block partitioning. K seeds in the clustering algorithm are automatically determined by using the hill-climbing method5 in the 3-D CIELab histogram of an input image, as shown in Fig. 2. Figure 2 describes the algorithms for the 1-D case. A resulting salient region is shown in Fig. 2. After labeling salient regions, the input image is partitioned into same-sized blocks, as shown in Fig. 2. These partitioned blocks generate the region of interest (ROI) map for the color shift, as shown in Fig. 2. We estimate the amount of color shift in the block of the ROI map by using the PCM proposed in Ref 6. The registered regions are shown in Fig. 2, and the finally fused image is shown in Fig. 2. Since the fused image can be considered a geometrically aligned version of the input image, we can further enhance the image by removing focusing blur as described in the following section.

Fig. 2

A sample procedure of the proposed CRF algorithm; (a) an input acquired by the MCA system, (b) the hill-climbing process, (c) the salient regions, (d) block partitioning, (e) new ROIs, (f) registered regions, and (g) the fused image.



Image Restoration for Enhancing the Multiple Color-Filter Aperture Output Image

We proposed a digital image restoration approach for overcoming fundamental limitations in the original MCA camera. We first estimate the PSF by assuming that the out-of-focus blur results in an isotropic or circularly symmetric PSF, and that the input image contains an ideal step edge at the linear boundary of an object. Using the estimated PSF,4 we generate the frequency response of the constrained least-squares (CLS) filter. The full-support spatial domain filter coefficients are obtained by the inverse Fourier transform of the CLS filter. Since significant energy is concentrated in the main lobe and its vicinity, we can have a smaller support spatial-domain filter by an appropriate truncation. Base on empirical optimization for both performance and efficiency, we adopted the raised-cosine function for the truncation window. A similar finite impulse response (FIR) restoration filter has been proposed for the extended depth of field (EDOF) system in the Bayer domain.7 In this work, we used the 13×13 truncated CLS (TCLS) filter with consideration for the size of the PSF.

Although the TCLS filter can remove out-of-focus blur with a limited amount of noise, there still exist undesired artifacts in the restored image. To completely remove such noise artifacts, we employ an additional space-adaptive mixing algorithm that takes detailed high-frequency regions from the restored image and takes flat regions from the input out-of-focus blurred image. For properly selecting one of two images, we compute the local variance at each pixel (x,y) as


where S represents a rectangular region encompassing (x,y) , f(x,y) is the pixel value at (x,y) , M and N are respectively the vertical and horizontal sizes of the region, and mxy is the local mean of the region. A general mixture of two images with weight α can be expressed as


where g represents the color-channel-aligned image by CRF, fR is the restored image by using the TCLS filter, and fA is the finally restored image by adaptive mixing. We compute the weighting factor as follows


where the tuning parameter σ is chosen so that α distributes as uniformly as possible in [0,1]. For the best normalized α , a typical value of σ is approximately equal to 1000. The alpha map image is produced by using Eq. 3. As a result, the finally restored image fA has sharply restored edges with suppressed noise.


Experimental Results

For evaluating the focusing performance of the proposed sets of algorithms, we used color images of size 1504×1000 acquired by the MCA system with a lens focal length of 150mm . The MCA output image, which contains multiple objects at different distances and the ROI map, is shown in Figs. 3 and 3, respectively. The result of CRF is shown in Fig. 3, where we note that the misalignment is corrected, while the out-of-focus blur still exists. Results of the TCLS restoration filter, the corresponding alpha map, and the alpha map-based SANS algorithm are respectively shown in Figs. 3, 3, 3.

Fig. 3

(a) The image acquired by the MCA system, (b) the ROI map image, (c) the shifted and aligned image by CRF, (d) the restored image by the TCLS filter, (e) the alpha map image, and (f) the result of the SANS.




The original version of the MCA system exhibits inherent limitations such as a manual CRF process and remaining focusing blur after CRF. For overcoming these problems, we propose a complete solution of real-time restoration incorporated into the MCA system in the form of the FIR filter structure. Furthermore, the proposed method can fully or partially be applied to a computational camera such as the MCA or EDOF system. Experimental results show that the proposed algorithm provides high visual quality by amplifying details of the image.


This work was supported by the Basic Science Research Program through the National Research Foundation (NRF) of Korea funded by the Ministry Education, Science, and Technology (2009-0081059).


1.  V. Maik, D. Cho, J. Shin, and J. Paik, “Regularized restoration using image fusion for digital auto-focusing,” IEEE Trans. Circuits Syst. Video Technol.1051-8215 17(10), 1360–1369 (2007). 10.1109/TCSVT.2007.903776 Google Scholar

2.  V. Maik, D. Cho, J. Shin, and J. Paik, “Color shift model-based segmentation and fusion for digital autofocusing,” J. Imaging Sci. Technol.1062-3701 51(4), 368–379 (2007). 10.2352/J.ImagingSci.Technol.(2007)51:4(368) Google Scholar

3.  Y. Bando, B. Chen, and T. Nishita, “Extracting depth and matte using a color-filtered aperture,” ACM Trans. Graphics0730-0301 27(5), 134:1–134:9 (2008). 10.1145/1409060.1409087 Google Scholar

4.  S. Kim, S. Park, and J. Paik, “Simultaneous out-of-focus blur estimation and restoration for digital auto-focusing system,” IEEE Trans. Consum. Electron.0098-3063 44(3), 1071–1075 (1998). 10.1109/30.713236 Google Scholar

5.  Z. Aghbari and R. Al-Haj, “Hill-manipulation: an effective algorithm for color image segmentation,” Elsevier Intl. J. Image Vision Comput. 24(8), 894–903 (2006). Google Scholar

6.  E. Castro and C. Morandi, “Registration of translated and rotated images using finite Fourier transforms,” IEEE Trans. Pattern Anal. Mach. Intell.0162-8828 9(5), 770–703 (1987). Google Scholar

7.  S. Kim, S. Jun, E. Lee, J. Shin, and J. Paik, “Real-time Bayer-domain image restoration for an extended depth of field (EDoF) camera,” IEEE Trans. Consum. Electron.0098-3063 55(4), 1756–1764 (2009). 10.1109/TCE.2009.5373728 Google Scholar

Sangjin Kim, Eunsung Lee, Vivek Maik, Joon-Ki Paik, "Real-time image restoration for digital multifocusing in a multiple color-filter aperture camera," Optical Engineering 49(4), 040502 (1 April 2010). https://doi.org/10.1117/1.3393885

Back to Top