1.

Introduction

Methods used in microscopic autofocus systems (MAFSs) are generally based on maximizing a focus measure obtained from a captured image; such measure is evaluated as a function of the lens’ $Z$ axis position. There are plenty of works that report algorithms to control the motion of the lens along the $Z$ axis in order to efficiently find the best position: based on the evaluation of that focus measure, they obtain the best focused single image.¹

In the case of a MAFS applied to cytology observation, when working with targets of high magnification—around $40\times$—one of the principal issues is that in many cases, cells are in fact located on different $Z$ levels in the slide, even corresponding to different ranges of the depth-of-field (DoF) of the lens. In these cases, the best focused image selected by a classical autofocus method will include unfocused parts, hence missing important information. A solution is to somehow combine a set of images captured with different DoFs to obtain a fully focused image.

Image fusion is the process of combining relevant visual information (i.e., important, complementary, and redundant) from multiple input images into a single resulting image. This should be achieved without introducing artifacts, in a way in which the resulting image contains more accurate, stable, and complete information^2–4 than the input images, therefore making it more suitable for human perception and for later processing operations (e.g., segmentation and feature extraction). For the autofocus application, this resulting image might be obtained by applying multifocus image fusion (MFIF) techniques: a set of $N$ images is captured from a static scene at different focus levels, and the focused objects in this set of images are fused together to create a sharp image with all those relevant objects fully focused.^1,3–5 While the reported MFIF methods are generally applied to fuse two images of a scene, the underlying techniques can be adapted to fuse a larger set of images, as we propose.

The context of our work is a project (see ACK section for details) to help early diagnosis of cervical intraepithelial neoplasia in the rural areas of the Coahuila State (Mexico). In these areas, for cultural reasons, the refusal among women to go to the capital persists until the symptoms of the disease are unbearable. The objective of the project is to enable the Papanicolau test, facilitating through its automation the taking of tissue samples in the rural area and the telematic sending of selected images of these samples for diagnosis by specialists. This requires capturing hundredths of focused images per tissue sample (see our autofocus contributions in Ref. 6) and analyzing of these images to identify and segment cervical nuclei (see our contributions on this area in Ref. 7). In this paper, we target the enhancement of the captured images via MFIF techniques.

MFIF techniques operate on a set of $N$ input images of a single scene, each with a different DoF. Overall, these images are first partitioned into generally homologous regions. Regions sharing a same location in the set of images are then evaluated, and the region with the highest focus measure is selected; finally, selected regions, usually from different images, are fused to compose the final focused image. There are many MFIF algorithms reported in the literature; a comprehensive review can be found in Refs. 8 and 9. The MFIF technique is broadly used in many application fields, such as microscopy,¹⁰ biology,¹¹ and medical imaging.¹²

In this paper, we analyze and compare with works using focus measures like those we used in our base autofocus system (i.e., transformed-domain measures⁶), for a fair comparison. In this direction, the work in Ref. 1 applies an $8\times 8$ block discrete cosine transform (DCT) to two input images; then, it compares homologous blocks of both images using the variance of the coefficients and selects the block with the highest value; finally, it applies the inverse DCT to the image composed by the selected blocks. They also propose a variant which applies a consistency verification index for block selection, in order to enhance the resulting image quality. The work in Ref. 4 is similar to that in Ref. 1 but using a different measure to compare blocks. Methods based on blocks generally present artifacts, because parts of the focused cells in different levels of the DoF might belong to a same block. Kumar⁵ uses the discrete cosine harmonic wavelet transform (DCHWT), a multiscale technique (three levels, in this case). As these multiscale methods involve decimation, most pixels in the resulting image do not keep original pixel values of any of the source images. Recently, some fusion techniques operating in the gradient domain have been reported, as that in Ref. 3. This work, also using a multiscale approach, uses a focus measure based on the saliency structure of the image, and it is designed to operate on well-known images (i.e., flower, clock, pepsi, etc.) with just two objects (one focused and the other unfocused).

Some recent works present similar fusion techniques applied to combine different sources of information into a single image but not necessarily due to a multifocus situation. Liu et al.¹³ described a MFIF method that separates source images into “cartoon content” and “texture content” via an improved iterative reweighted decomposition algorithm; fusion rules are designed to separately fuse both types of content, and finally, the fused cartoon and texture components are combined. The technique naturally approximates the morphological structure of the scene. The work in Ref. 14 presents a medical application to diagnose vascular diseases; they use a type of wavelet transform combined with an averaging-based fusion model to fuse osseous and vascular information together; they present a rapid MFIF algorithm, less complex but still very effective, with very low memory requirements. For a similar objective, Dogra et al.¹⁵ propose an effective image fusion method also working on the wavelet domain, along with a preprocessing of the source images with a selected sequence of spatial and transformed-domain techniques to create a highly informative fused image for osseous-vascular 2-D data.

In this paper, we propose an MFIF method that analyzes sequences of up to 15 microscopy input images corresponding to different levels of DoF of a same “slide-scene.” We propose (Sec. 2) an object-based approach, which dramatically reduces the visibility of fusion-generated artifacts while keeping focused parts of the input images intact. To evaluate our results, we compare with five different existing techniques (Sec. 3) by testing over 50 realistic and practical Pap smear sequences of images, and over the two blurred microscopic images provided by Ref. 11. Finally, conclusions are presented in Sec. 4.

2.

Proposed Multifocus Microscope-Image Fusion Method

Figure 1 illustrates the general flow of the proposed MFIF method, which is further detailed in the following subsections. The starting point is a set of images (15 in our experiments, but 2 in other works we compare to) captured with the lens in a varying position of the $Z$ axis. The first step, which is not the topic of this paper, is the selection of the “best-focused image” of the set (see details in Ref. 6). Let $\{I_i,i=1\dots N\}$ be this set of input images (Fig. 1) and let $I_{\mathrm{bf}}$ be the best-focused image, being $i=bf$ its index in the set. This image is first coarsely segmented to identify its main regions or objects, which are considered the main scene objects, each represented by a binary mask. Then, for each scene object or segmented region, its mask is applied to the set of input images, and a focus measure is obtained for that region in every image of the set. A preliminary image, which we name “combined image, $\mathrm{I}\_\mathrm{c}$,” is then generated by replacing in the best-focused image each segmented region with the corresponding best-focused region in the set. Finally, the removal of the artifacts generated in the contours of these combined regions is performed by a total variational-based filter¹⁶ to obtain the “final focused image, $\mathrm{I}\_\mathrm{ff}$.”

Fig. 1

Proposed multifocus image fusion method for a set of microscopic high-magnification Pap smear images.

2.2.

Preliminary Image Fusion Based on the DCT Focus Measure

The next step is to generate a “combined image” (follow this process in Fig. 2), $\mathrm{I}\_\mathrm{c}$, which is a merging of the best focused parts of the set of input images. First, a focus measure is locally obtained for every image of the set, following the method described in Ref. 6: in brief, for every image of the input set, $I_i$, a $8\times 8$ block DCT is performed, the sum of the absolute value of its 32 lower-frequency AC coefficients is calculated, and a same-size energy image, $E_i$, is obtained by assigning each pixel the calculated energy of its corresponding block. This results in a set of DCT energy images, $\{E_i,i=1\dots N\}$ [Fig. 2(a)].

Fig. 2

Obtaining the combined image, $I\_c$ form the set of input images, $\{I_i,i=1,\dots ,N\}$.

The topology of the $\mathrm{I}\_\mathrm{c}$ image is the same of that of the segmented image, $\mathrm{I}\_\mathrm{seg}$. Every cluster or region in $\mathrm{I}\_\mathrm{seg}$ is used to generate a mask, $\{M_r,r=1\dots N_c\}$ [Fig. 2(b)]. For every region, its corresponding mask is applied over every energy image, $\{E_i,i=1\dots N\}$, and the mean energy of the masked region is calculated for every such energy image. The index, $i\_\max$, of the energy image showing maximum energy for that region is obtained; then, the corresponding region of the $\mathrm{I}\_\mathrm{c}$ image is initialized with the pixel values of the homologous region of the $I_{i\_\max }$ image from the $\{I_i,i=1\dots N\}$ set. In parallel, in the $\mathrm{I}\_\mathrm{diff}$ image (see Fig. 2), we keep for every region the absolute difference between the $\mathrm{i}\_\max$ index and the $i=\mathrm{bf}$ index of the best-focused image, which somehow indicates the degree of out-of-focus of such region, or the object focus level, ranging from 0 (black-level: the region is best-focused in the $I_{\mathrm{bf}}$ image) to $N$ (white-level: the region is best-focused in the image with the worst global focus measure).

As opposed to other methods, such as Refs. 1 and 4, where the local focus comparison among the set of input images is performed block by block, we propose to compare region by region using the segmentation of the best-focused image to define such regions. This avoids highly visible block artifacts appear anywhere. Instead, contour artifacts might appear in the boundaries of the identified regions, being here much less visible. The visibility of these contour artifacts depends on the aforementioned degree of out-of-focus of each combined region. In the next subsection, we propose to use a total variational-based filter to eliminate the contour artifacts of the combined image, $\mathrm{I}\_\mathrm{c}$.

2.3.

Artifacts Removal

The next step is to generate the final focused image, $\mathrm{I}\_\mathrm{ff}$, by attenuating the artifacts or false contours that may appear in the combined image, $\mathrm{I}\_\mathrm{c}$, due to merging regions from different input images. We propose to attenuate artifacts by applying a total variational-based diffusion method.¹⁶ This method will only be applied in the artifacts-prone areas, according to the information in the $\mathrm{I}\_\mathrm{diff}$ image, hence preserving or keeping intact most of the image pixels. Observe that the method aims to mitigate these false contours, not real object contours.

2.3.1.

Generation of a mask of the artifact-prone areas

In Fig. 3, we show several examples of artifacts generated at the boundaries of the regions of three $\mathrm{I}\_\mathrm{c}$ images. Our proposal is to process $\mathrm{I}\_\mathrm{c}$ pixels only at the edges defined by $\mathrm{I}\_\mathrm{diff}$, i.e., only at the boundaries between regions with different degree of focus, in order to obtain an image without artifacts on these boundaries while keeping original pixels in most of the resulting image. For this purpose, we first obtain an edges image from $\mathrm{I}\_\mathrm{diff}$ (see Fig. 4). Then, as the extent of the artifacts between adjacent regions is expected to be proportional to the difference between their degrees of focus, we perform an adaptive morphological dilation over the thresholded edges image, using a structural element with a size proportional to the intensity of every edge. The resulting mask, $\mathrm{M}\_\mathrm{artifacts}$ (see Fig. 4), will define where the following enhancement steps will be applied.

Fig. 3

Examples of contour artifacts in the boundaries of the merged regions: (a) $I\_\mathrm{diff}$; (c) $I\_c$; (b) and (d) detailed images of the artifacts (see white arrows) generated by the regions merging.

Fig. 4

Example of the generation of the artifacts mask: (a) $I\_c$, (b) $I\_\mathrm{diff}$, (c) edges in $I\_\mathrm{diff}$, and (d) dilated and thresholded edges in $M\_\text{artifacts}$.

A main contribution of our method is that artifacts removal is only performed in the areas that may include them, hence preserving original pixels in most of the image, which is critical for medical imaging applications. Works in Refs. 1 and 4, as they perform image fusion over DCT blocks, are prone to generate block-artifacts, which are not later eliminated. In the multiscale methods, such as Refs. 5 and 3, the original pixels are not usually preserved in the fused image: the resulting image in Ref. 5 does not present artifacts due to the nature of the method, which modifies pixels intensity via averaging, resulting in a smoother image; the method proposed in Ref. 3 eliminates artifacts just in the “unknown zone,” which is a predefined area in the boundary generated between the two considered source images with two different focus levels.

2.3.2.

Artifacts removal via total-variation filtering

Let us consider that the combined image $\mathrm{I}\_\mathrm{c}$, which contains contour artifacts in the contours defined by the $\mathrm{M}\_\text{artifacts}$ mask, is a noisy image; let $\mathrm{I}\_\mathrm{ff}$, the final focused image, be the desired sharp and clean image. We can then declare that $\mathrm{I}\_\mathrm{c}=\mathrm{I}\_\mathrm{ff}+n$, where $n$ is the aditive noise, which we assume concentrated in the pixels indicated by $\mathrm{M}\_\text{artifacts}$. We obtain $\mathrm{I}\_\mathrm{ff}$ from $\mathrm{I}\_\mathrm{c}$ using a total variation filter. These filters were first suggested by Ref. 16 and are based on the minimization of an energy functional, subject to the constraint $u-u_0={\sigma }^2$, where $u$, $u_0$, and ${\sigma }^2$, for this work, are, respectively, the gradient of the true image ($\mathrm{I}\_\mathrm{ff}$), the gradient of observed image ($\mathrm{I}\_\mathrm{c}$), and the variance of the noise $n=(u-u_0)$. Then, the iterative equation to obtain the desired clean image is $\mathrm{I}\_{\mathrm{ff}}^t=\mathrm{I}\_{\mathrm{c}}^t-\lambda (u-u_0^t)$, where $\mathrm{I}\_{\mathrm{c}}^0=\mathrm{I}\_\mathrm{c}$, and $\lambda$ is a regularization parameter which we set to $\lambda =0.1$ to preserve the smallest structures.

In order to obtain $u_0$ for the first iteration, we apply a Laplacian filter to $\mathrm{I}\_\mathrm{c}$ (Fig. 5). To estimate $u$, we consider that its gradient equals that of $\mathrm{I}\_\mathrm{c}$ except for the edge areas defined by $\mathrm{M}\_\text{artifacts}$. In these areas, for every pixel, we assume that $u$ equals the gradient, $\{u_i,i=1\dots N\}$, of the source image showing maximum local variance around such pixel. The gradients of the source images are also obtained applying a Laplacian filter, and the local variance is computed in $3\times 3$ windows. Once we get $\mathrm{I}\_{\mathrm{ff}}^t$ for this first iteration, i.e., $\mathrm{I}\_{\mathrm{ff}}^0$, we set $\mathrm{I}\_{\mathrm{c}}^{t+1}=\mathrm{I}\_{\mathrm{ff}}^t$ and repeat the process until it converges to $\mathrm{I}\_\mathrm{ff}$. Figure 5 shows an example of the evolution of the variance of the gradient difference, $u-u_0^t$, and of the obtained image, $\mathrm{I}\_{\mathrm{ff}}^t$, for every iteration.

Fig. 5

Example of the iterative artifacts elimination process: (a) evolution of the gradients difference $u_0^t-u$ and (b) comparison of the $I\_c$ image and of the $I\_ff$ image obtained after the process converges; white arrows indicate the main removed artifacts.

3.

Experimental Results

To assess the potential of our approach, we compare the proposed method with the works reported in Refs. 1-1, 1-2, 5, 3 and 4, as we can see in Figs. 6–8 captions, and Tables 1 and 2. The code to run these reported algorithms was kindly provided by every author: the software is available together with the papers. The set of images used for the experimental evaluation, hereinafter the dataset, consist of the microscopic image pair from the MFIF reported in Ref. 11 (see Fig. 6) and a set of 50 Pap smear image sequences ($320\times 240\text{pixels}$ in RGB), each containing 15 images with different DoF and focused cells in several of them (see examples in Figs. 7 and 8).

Fig. 6

Visual results for the first experiment. The top row shows full images and the bottom one shows the corresponding image in detail. Columns include source images A (I) and B (II); resulting images for the compared methods 1-1 (a) Haghighat, 2011(1), 1-2 (b) Haghighat, 2011(2), 5 (c) Kumar, 2013, 3 (d) Zhou, 2014 and 4 (e) Phamila, 2014; (f) the resulting image, $\mathrm{I}\_\mathrm{ff}$.

Fig. 7

Visual results for the second experiment. The top row shows the sequence of input images for sequence 3 (I). The bottom two rows show full images and the corresponding image detail for the final fused images obtained by each compared method: 1-1 (a) Haghighat, 2011(1), 1-2 (b) Haghighat, 2011(2), 5 (c) Kumar, 2013, 3 (d) Zhou, 2014, 4 (e) Phamila, 2014 and (f) the resulting image, $\mathrm{I}\_\mathrm{ff}$.

Fig. 8

Visual results for the second experiment. The top row shows the sequence of input images for sequence 37 (I). The bottom two rows show full images and the corresponding image detail for the final fused images obtained by each compared method: 1-1 (a) Haghighat, 2011(1), 1-2 (b) Haghighat, 2011(2), 5 (c) Kumar, 2013, 3 (d) Zhou, 2014, 4 (e) Phamila, 2014 and (f) the resulting image, $\mathrm{I}\_\mathrm{ff}$.

3.2.

Experiments and Discussion

The first experiment is conducted over two microscopic images kindly provided by Ref. 11, each showing different focused parts of the same object [see Figs. 6(I) and 6(II)]. We have applied to these source images the aforementioned five fusion algorithms and our proposed method. Figure 6 shows the resulting images along with a detail of each, in order to visually or qualitatively assess the performance of each method. Table 1 includes data with the quantitative evaluation of this first experiment.

Table 1

Quantitative results for the first experiment: performance quality metrics for the final fused microscope image obtained by each method.

	1-1 Fig. 6 (a) Haghighat, 2011(1)	1-2 Fig. 6 (b) Haghighat, 2011(2)	5Fig. 6 (c) Kumar, 2013	3Fig. 6 (d) Zhou, 2014	4Fig. 6 (e) Phamila, 2014	I_ffFig. 6 (f)
SD	0.9877	0.9783	0.9534	0.9957	0.9843	1
$Q^{AB/F}$	0.8605	0.7629	0.8829	0.8884	0.8436	0.8919

From a qualitative point of view, we observe that methods Refs. 1-1 and 4 [Figs. 6(a) and 6(e)] present highly visible block artifacts; these methods compare the DCT energy in homologous $8\times 8$ blocks, which generates comparison errors when the images contain nonsquare elements in different depths of field or when cervical cells are round. The enhancement proposed by Ref. 1-2 [Fig. 6(b)], based on a consistency verification index to decide which block is selected, removes block artifacts in this example but at the expense of a poor visual result. The multiscale approach proposed in Ref. 5 [Fig. 6(c)], which does not keep original pixel values, presents a noticeable contrast reduction. The method reported in Ref. 3 and the proposed method [Figs. 6(d) and 6(f)] yield acceptable visual results.

From a quantitative point of view (see Table 1), an interesting observation is to contrast the correlation between each measure of quality and the perceived visual result: the SD measure yields very good values for images with highly noticeable block artifacts [in case of Figs. 6(a) and 6(e)], because these artifacts increase image variance; the $Q^{AB/F}$ measure seems to be more in line with the qualitative findings.

Independently of these observations, Table 1 indicates that the proposed method behaves better in the light of both quality measures.

The second experiment targets the 50 sequences of Pap smear images obtained from the autofocus operation of a microscope. While reported works have focused on fusing two blurred images, many of them have applied their method in an iterative way to more than two input images, which is our practical context. Figures 7 and 8 show qualitative results for two of these sequences, and Table 2 and Fig. 9 compile the quantitative evaluation for the 50 sequences.

Table 2

Quantitative results for the second experiment: performance quality metrics (mean and deviation) for the final fused microscope images obtained by each method applied to the 50 image sequences.

	1-1 Haghighat, 2011(1)	1-2 Haghighat, 2011(2)	5 Kumar, 2013	3 Zhou, 2014	4 Phamila, 2014	I_ff
SD	$0.9632\pm 0.0187$	$0.9832\pm 0.0134$	$0.8397\pm 0.0726$	$0.9424\pm 0.0359$	$0.9785\pm 0.0106$	$0.9987\pm 0.0045$
$Q$	$0.9920\pm 0.0072$	$0.9897\pm 0.0152$	$0.8941\pm 0.1011$	$0.6630\pm 0.0678$	$0.9940\pm 0.0081$	$0.9974\pm 0.0041$

Fig. 9

Quantitative results for the second experiment. Performance quality metrics SD and $Q$ for the final fused microscope images obtained by each method for every image sequence.

From a qualitative point of view, we clearly observe in Fig. 7 that the methods based on $8\times 8$ DCT blocks [Figs. 7(a), 7(b) and 7(e), 8(a), and 8(e)] cannot avoid generating block artifacts. We can also observe that the multiscale approaches [Figs. 7(c), 7(d), 8(c), and 8(d)], which somehow process original pixels so that their value is never directly transferred to the final image, suffer from a severe loss of definition when the technique is applied to a large number of source image (15 images, instead of 2, for this experiment): several of the objects of interest are averaged, resulting in a loss of information and even the loss of complete cells. This is the situation for the method proposed in Ref. 3 [Figs. 7(d) and 8(d)], which, while losing very few information and objects of interest, sometimes loses full objects because it only compares two areas or regions in the image (focused and unfocused).

From a quantitative point of view, Table 2 indicates that the proposed method also behaves better for this part of the dataset including 50 image sequences. Apart from the mean values of the quality indicators, Table 2 includes their standard deviation, which proves that the results obtained by the proposed method are also the most stable. Finally, Fig. 9 intends to further illustrate the stability of the tested methods that obtained better global results. We observe that the proposed method systematically outperforms other approaches in the light of these quality indicators.

4.

Conclusion

This paper presents an object-oriented approach to the problem of obtaining a single focused image from a set of microscopic images captured from a single slide including objects that happen to be focused each in a different image of the set. The proposed MFIF method shows several specific advantages respect to other state-of-the-art methods: first, it is driven by a region-based segmentation, which prevents for the highly visible artifacts that may appear in block-based methods; second, it does not apply any kind of image transform, hence respecting the pixel-values of all focused regions, which is crucial for medical imaging applications; and finally, it includes a artifacts-removal technique, which only operates were required and adapts to the expected extent of the fusion-generated artifacts. Results, obtained over a representative dataset and compared to other published approaches, prove the validity of our proposal.