Computational cameras with multifocusing functions have combined optics and signal processing techniques to overcome limitations in a pure digital signal processing approach, such as in Ref. 1. The multiple color-filter aperture (MCA)2, 3 system is one such computational camera technique, by which a single camera system can estimate depth information of objects at different distances and perform multifocusing. The existing MCA system, however, exhibits several problems, including low exposure, color mixing, deviation of color convergence, and divergence of light rays. These problems result in nonideal point-spread functions (PSFs), which deteriorate the overall quality of the image even after color channel registration and fusion (CRF). In this work, we present: 1. fundamental principles of the MCA computational camera, 2. the automatic CRF process, and 3. a real-time image restoration system for improving the quality of images acquired by the MCA system. For the image restoration process, we first estimate the PSF based on the method proposed in Ref. 4, and then design the truncated constrained least-squares (TCLS) restoration filter using the estimated PSF. Most image restoration filters tend to amplify noise and produce ringing artifacts, and the proposed TCLS filter is not an exception. For removing such artifacts, we additionally propose a spatially adaptive noise smoothing (SANS) algorithm using adaptively weighted mixing between the color-channel-aligned image and the restored image by the TCLS filter.
Automatic Color Channel Registration and Fusion for the Multiple Color-Filter Aperture Camera
The aperture of an optical system is the opening device that adjusts the amount of incoming light. Generally, the center of an aperture is aligned to the optical axis of the lens, and the convergence pattern on the imaging plane forms either a point or a circular region. However, if the center of the aperture is not aligned to the optical axis, the image formation of the sensor deviates from the optical axis. As shown Fig. 1, the MCA system consists of three apertures with R, G, and B color filters. The main advantage of the MCA system is that it provides additional depth information of objects at different distances, as shown in Figs. 1, 1, 1. At the cost of additional depth information, the image acquired by the MCA system contains a certain amount of out-of-focus blur, which results from color misalignment. For solving the color misalignment problem, manual registration of color channels and fusion methods have been proposed in Refs. 2, 3.
We propose cluster-based image segmentation and phase correlation matching (PCM) for automatically registering pixels in three color channels. The proposed cluster-based segmentation algorithm employs K-means clustering and block partitioning. K seeds in the clustering algorithm are automatically determined by using the hill-climbing method5 in the 3-D CIELab histogram of an input image, as shown in Fig. 2. Figure 2 describes the algorithms for the 1-D case. A resulting salient region is shown in Fig. 2. After labeling salient regions, the input image is partitioned into same-sized blocks, as shown in Fig. 2. These partitioned blocks generate the region of interest (ROI) map for the color shift, as shown in Fig. 2. We estimate the amount of color shift in the block of the ROI map by using the PCM proposed in Ref 6. The registered regions are shown in Fig. 2, and the finally fused image is shown in Fig. 2. Since the fused image can be considered a geometrically aligned version of the input image, we can further enhance the image by removing focusing blur as described in the following section.
Image Restoration for Enhancing the Multiple Color-Filter Aperture Output Image
We proposed a digital image restoration approach for overcoming fundamental limitations in the original MCA camera. We first estimate the PSF by assuming that the out-of-focus blur results in an isotropic or circularly symmetric PSF, and that the input image contains an ideal step edge at the linear boundary of an object. Using the estimated PSF,4 we generate the frequency response of the constrained least-squares (CLS) filter. The full-support spatial domain filter coefficients are obtained by the inverse Fourier transform of the CLS filter. Since significant energy is concentrated in the main lobe and its vicinity, we can have a smaller support spatial-domain filter by an appropriate truncation. Base on empirical optimization for both performance and efficiency, we adopted the raised-cosine function for the truncation window. A similar finite impulse response (FIR) restoration filter has been proposed for the extended depth of field (EDOF) system in the Bayer domain.7 In this work, we used the truncated CLS (TCLS) filter with consideration for the size of the PSF.
Although the TCLS filter can remove out-of-focus blur with a limited amount of noise, there still exist undesired artifacts in the restored image. To completely remove such noise artifacts, we employ an additional space-adaptive mixing algorithm that takes detailed high-frequency regions from the restored image and takes flat regions from the input out-of-focus blurred image. For properly selecting one of two images, we compute the local variance at each pixel asrepresents a rectangular region encompassing , is the pixel value at , and are respectively the vertical and horizontal sizes of the region, and is the local mean of the region. A general mixture of two images with weight can be expressed as represents the color-channel-aligned image by CRF, is the restored image by using the TCLS filter, and is the finally restored image by adaptive mixing. We compute the weighting factor as follows is chosen so that distributes as uniformly as possible in [0,1]. For the best normalized , a typical value of is approximately equal to 1000. The alpha map image is produced by using Eq. 3. As a result, the finally restored image has sharply restored edges with suppressed noise.
For evaluating the focusing performance of the proposed sets of algorithms, we used color images of size acquired by the MCA system with a lens focal length of . The MCA output image, which contains multiple objects at different distances and the ROI map, is shown in Figs. 3 and 3, respectively. The result of CRF is shown in Fig. 3, where we note that the misalignment is corrected, while the out-of-focus blur still exists. Results of the TCLS restoration filter, the corresponding alpha map, and the alpha map-based SANS algorithm are respectively shown in Figs. 3, 3, 3.
The original version of the MCA system exhibits inherent limitations such as a manual CRF process and remaining focusing blur after CRF. For overcoming these problems, we propose a complete solution of real-time restoration incorporated into the MCA system in the form of the FIR filter structure. Furthermore, the proposed method can fully or partially be applied to a computational camera such as the MCA or EDOF system. Experimental results show that the proposed algorithm provides high visual quality by amplifying details of the image.
This work was supported by the Basic Science Research Program through the National Research Foundation (NRF) of Korea funded by the Ministry Education, Science, and Technology (2009-0081059).