Bio-empirical mode decomposition: visible and infrared fusion using biologically inspired empirical mode decomposition

Paterne S. Sissinto; Jumoke O. Ladeji-Osias

doi:10.1117/1.OE.52.7.073101

8 July 2013 Bio-empirical mode decomposition: visible and infrared fusion using biologically inspired empirical mode decomposition

Paterne S. Sissinto, Jumoke O. Ladeji-Osias

Author Affiliations +

Optical Engineering, Vol. 52, Issue 7, 073101 (July 2013). https://doi.org/10.1117/1.OE.52.7.073101

Abstract

Bio-EMD, a biologically inspired fusion of visible and infrared (IR) images based on empirical mode decomposition (EMD) and color opponent processing, is introduced. First, registered visible and IR captures of the same scene are decomposed into intrinsic mode functions (IMFs) through EMD. The fused image is then generated by an intuitive opponent processing the source IMFs. The resulting image is evaluated based on the amount of information transferred from the two input images, the clarity of details, the vividness of depictions, and range of meaningful differences in lightness and chromaticity. We show that this opponent processing-based technique outperformed other algorithms based on pixel intensity and multiscale techniques. Additionally, Bio-EMD transferred twice the information to the fused image compared to other methods, providing a higher level of sharpness, more natural-looking colors, and similar contrast levels. These results were obtained prior to optimization of color opponent processing filters. The Bio-EMD algorithm has potential applicability in multisensor fusion covering visible bands, forensics, medical imaging, remote sensing, natural resources management, etc.

1. Introduction

Image fusion is the process of combining two or more registered images of the same scene to get a more informative image. Visible and infrared (IR) color image fusion has become a process with multiple applications. From situational awareness to medical imaging, fusion has provided users with images that are more meaningful than source images. Image fusion techniques can be broken down into two main approaches: multiscale and nonmultiscale.¹ Multiscale techniques include wavelet transforms and pyramid transforms.² Nonmultiscale techniques include linear, nonlinear, estimation theory, artificial neural networks, and color composite approaches.² Despite, the enormous research done on the subject, obtaining a fused image with very high information content and an informative depiction of the scene is a domain of active research. Several image fusion techniques have been developed by researchers over the last two decades to perform the synergistic combination of different sensory information at the pixel, feature, or decision level.³^–⁶ Most of these methods perform the integration of grayscale images. Since human eyes distinguish about a hundred grayscale levels and thousands of color variations, color-fused images provide more information content than grayscale fusion. Color fusion provides a chromatic representation of the fused image in false or near-true colors for situational awareness and medical applications. Some studies were done on fusing a thermal IR image with electro-optical (EO) or visible images; false colors were obtained.⁷^–⁹ These works point out three gaps: (1) there is a need to develop and implement efficient algorithms performing color fusion of IR and EO images and generating natural-looking colors; (2) the systems need to transfer as much information as possible from the input images to the fused one, and generate a high-quality image, scalable in terms of number and size of images being integrated; and (3) in information fusion domain, empirical mode decomposition (EMD) is a fully data-driven technique that provides a decomposition of images into finite sets of signals called intrinsic mode functions (IMFs) and the literature points out that better fusion can be achieved on IMFs.

In this work, we proposed a new technique to fuse low-light visible and IR images and generate near naturally looking colors. The method is based on EMD and center-surround opponent processing.¹⁰^,¹¹ This paper is organized as follows. EMD, opponent-processing, and image fusion quality metrics, which form the theoretic background, are introduced in Sec. 2. In Sec. 3, we present the structure and algorithm supporting this work. Section 4 is based on the result evaluation and comparison to some existing technique outcome for some sets of images. Finally, conclusions are drawn in Sec. 5.

2. Background and Concepts

In this section, we present the necessary theoretical background for the development of an opponent processing information fusion technique. This includes the dynamic neural network equations and techniques developed to produce color-fused images.

2.1.

Dynamic Neural Network Equations

Visible electromagnetic waves have a wavelength between 400 (violet) and 700 nm (red). Human eyes are more sensitive to colors in the middle of the visible spectrum (green to yellow) and have dim sensitivity of the spectrum toward the extremes.⁷ Like several nature-inspired solutions, the human vision system has inspired some image processing developments. Inside the retina, photoreceptors are responsible for image formation. Rods participate in achromatic image formation which has poor details and no color. Conversely, cones, horizontal cells, and bipolar cells produce contrast enhancement of color information which correspond to spatial opponent signal processing. There are three types of cones containing photo-pigments with distinct spectral sensitivity.² $L$ cones are sensitive to long-wavelengths around 560 nm, $M$ cones are sensitive to medium-wavelengths around 530 nm, and $S$ cones are sensitive to short-wavelengths around 420 nm. These cones share some sensitivity regions. Emulating retina processing has resulted in image fusion architectures with center-surround operations.¹²^,¹³

Center-surround operations are the result of cones transforming photons into signals through opponent mechanisms. The relations governing the activities in the retina when the excitation and the inhibition are performed by filtering (center-surround operations)² are summarized in Table 1, which illustrates the channels within the luminance and color that are coupled within the retina, resulting in one being excited and the other being inhibited.

Table 1

Retina color excitation and inhibition.

Photoreceptors	Signals coupled	Excited	Inhibited
Rods	Light–Dark	Light	Dark
Cones	Blue–Yellow	Blue	Yellow
Cones	Red–Green	Red	Green

The neuro-dynamic interactions representing the center-surround model at a pixel ( $x_{i j}$ ) level are summarized in Table 2.

Table 2

ON/OFF center-surround filtering.

	ON-center	OFF-center
Equation	$\frac{d}{d t} x_{i j} = - A x_{i j} + (B - x_{i j}) C_{i j} + (x_{i j} + D) E_{i j}$	$\frac{d}{d t} {\bar{x}}_{i j} = - A ({\bar{x}}_{i j} - S) + (\bar{B} - {\bar{x}}_{i j}) {\bar{C}}_{i j} + ({\bar{x}}_{i j} + \bar{D}) {\bar{E}}_{i j}$
Filters	$C_{p q i j} = C \exp {- α^{- 2} \log 2 [{(p - i)}^{2} + {(q - j)}^{2}]}$	$E_{p q i j} = E \exp {- β^{- 2} \log 2 [{(p - i)}^{2} + {(q - j)}^{2}]}$
Filtered	$C_{i j} = \sum_{p, q} I_{p q} C_{p q i j}$	$E_{i j} = \sum_{p, q} I_{p q} E_{p q i j}$
Images	$x_{i j} = \frac{\sum_{(p q)} (B C_{p q i j} - D E_{p q i j}) I_{p q}}{A + \sum_{(p q)} (C_{p q i j} + E_{p q i j}) I_{p q}} = x_{i j}^{+}$	${\bar{x}}_{i j} = \frac{A S + \sum_{(p q)} (D E_{p q i j} - B C_{p q i j}) I_{p q}}{A + \sum_{(p q)} (C_{p q i j} + E_{p q i j}) I_{p q}} = x_{i j}^{-}$

Note: S=.2, B¯=D, C¯ij=Eij, D¯=B, and E¯ij=Cij

$A = 134$ , $B = 1$ , $C = 7$ , $D = .5$ , $E = 3.333$ , $α = 1.3$ , and $β = 1.875$ such that both Gaussian kernel filters cover the same area.²^,¹¹ $C_{i j}$ is the ON-center interaction and $E_{i j}$ is the OFF-center interaction; both represent a discrete convolution of the input pattern $I_{i j}$ with a Gaussian kernel. At the equilibrium, $x_{i j}$ has a constant value so its derivative is equal to zero. The coefficient $A$ affects the lightness/darkness of the filtered image.

2.2.

Color Fusion of IR and Visible Images

Unlike grayscale image fusion, color fusion provides a chromatic representation of the fused image in false or real color. Figure 1 shows a hierarchical model of some color composite image fusion procedures, their authors, and institutions. In many cases, image fusion approaches are applied in combination with other algorithms.

Fig. 1

Color composite image fusion techniques.

Toet et al. developed a false color mapping technique where the “unique” and “common” components of two images are assigned to the RGB band.⁷^,¹⁴ Their results showed enhancement of features unique to each modality. However, common features were diminished in the fused image and resulted in colors that were different from the original color image. Remapping different gray levels of a unique region in images produced different colors, a process that creates unsatisfactory color visual effects.

Waxman et al.¹⁵^–¹⁸ developed a variety of low-light visible/IR fusion architectures that merge EO images with thermal IR imagery by emulating some principles of biological opponent-color vision. Their approach to frame fusion relied on biologically motivated neuro-computational models of visual contrast enhancement. Their architecture fused EO image and thermal images successfully, but the integrity of the color information is not preserved, reducing the ability to recognize objects. This was the case of the other architectures developed by this team during the same study.

Relying on Land’s experiment on color constancy of human vision,¹⁹ Huang et al proposed a new method to fuse visual and IR images and generate a false color image. Their proposed architecture is based on equal energy distribution assumption of colors reflected to eyes. Testing results showed lower colorfulness compared to Toet et al. methods but allowed target detection.¹⁹^,²⁰ These results also confirmed that the reddish features in the fused image are pulled from the IR source while greenish objects are from the visible source.

Nunez et al.²¹ developed a new approach to merge high-resolution panchromatic images with low resolution-multispectral images. Several techniques offer the conversion of multispectral images into intensity hue saturation (IHS).²²^,²³ This method has the advantage of adding the spectral quality of the color image to high resolution details from the panchromatic image. Similar frameworks have been applied utilizing pyramid-based fusion method.²⁴ An expansion of this work applying the spectral response of sensors is detailed in Ref. 22.

2.3.

Evaluation of Color Fusion

Fusion evaluation metrics have been largely developed for still images. Fused frames are evaluated on accuracy, robustness, and sensitivity of the generating algorithms. Image fusion can be done subjectively or objectively. In subjective image evaluation, an audience of qualified observers grade the results of integration based on the amount of useful information extracted from the original images. Conditions of observation must be identical for all observers and the screen must be sufficiently large.²⁵^,²⁶ This work utilized concepts developed to assess results from fusion processes objectively.⁹^,²⁷^–³¹

2.4.

Empirical Mode Decomposition

EMD¹⁰^,³²^–³⁴ is a nonparametric and self-adaptive method which makes effective use of an image data to derive its decomposition into a set of finite IMFs. This is important when fusing real-world images. The advantages of EMD are multiple. EMD is self-adaptive, nonparametric, make no assumption about data being decomposed and corresponds to the nonstationary and nonlinear behavior of imagery from different modalities. Also, this approach is computationally light and intuitive compared to other decomposition techniques.¹⁰^,³⁵ Figure 2 shows how the IMFs are generated.

Fig. 2

Empirical mode decomposition flow chart.

The image to be decomposed is first converted from two-dimensional (2-D) array to one-dimensional (1-D) array and treated as a signal $x (t)$ (function of $t$ where $t$ goes from 1 to the number of pixels in the image). The colors of the pixels determine the amplitude of the signal at each index. The maxima and minima of $x (t)$ are identified and generate the upper and lower splines (envelopes). The mean signal $m (t)$ of these two envelopes is subtracted from the original signal $x (t)$ to obtain a new signal $h (t)$ . If $h (t)$ is symmetric to the zero-crossing axis and the difference between the number of maxima and minima is not greater than 1, $h (t)$ is considered an IMF and $x (t)$ is replaced by the residual $x (t) - h (t)$ ; otherwise $x (t)$ is replaced by $h (t)$ . The process stops when $r (t)$ becomes monotonic. The EMD provides a decomposition of the images into IMFs and residual. Each IMF sample carries pixel color information. For original images of size $M \times N$ , each IMF will be $M * N$ samples long; the information is stored as row vectors. Depending on the image content and the sifting process utilized, the decomposition generates a certain number $T$ of IMFs in total.

3. Bio-EMD Fusion of IR and Visible Images

Integration of IR and visible images should generate a fused imagery with a high level of information transferred, present clarity of details, facilitate detection or identification, and render near true colors. Our approach is to fuse the spatial and frequency components of the input images obtained by EMD through opponent processing. Figure 3 presents the design model.

Fig. 3

Design model.

The source images (IR and visible) are registered images. The IR image is grayscale but may also be a dual band or RGB signal, depending on the sensor output. The visible image is a color image. Both images are pre-processed for noise removal, contrast enhancement, and resized to the same size (numbers of rows and columns), if different. These source images are decomposed into their IMFs, generated through EMD³³^,³⁵^,³⁶ according to the architecture presented in Fig. 4 before the fused-image reconstruction process (sum of IMFs and conversion from 1-D to 2-D).

Fig. 4

IMFs fusion in Bio-EMD.

Figure 3 presents the developed IMF integration model. The IMFs are integrated following a biological model that emulates the human retinal system. The red, green, and blue signals of the color image are decomposed into their respective IMFs. The IR image is converted in luminance (Y), chrominance blue (Cb), and chrominance red (Cr). The chrominance is not utilized in the model but may convey temperature information when added to the fused IMFs. EMD provides a decomposition of the images into IMFs and residual. Each IMF sample carries pixel color information. For original images of size $M \times N$ , each IMF is $M * N$ samples long; the information is stored as row vectors. In the retina, there are three types of cones containing photo-pigments having distinctive spectral sensitivity¹¹ and signals are treated in pairs black–white, blue–yellow, and red–green to create all color perceptions. In this model, the yellow signal is synthesized by combining the red and yellow signals out of the RGB image. These sensitivities are expressed in images as excitation and inhibition of representing signals. The excitation process, called ON-center, enhances/enlightens the image; the inhibition process or OFF-center depletes/darkens the image. In Fig. 3 the sign “+” represents ON center-surround process and “−” indicates OFF center-surround filtering.

For an IMF of rank $k$ ( $1 \leq k \leq T$ ) the ON-center IMF, $F$ is computed as

Eq. (1)

F_{k} (i) = \frac{{Num}_{k} (i)}{{Den}_{k} (i)}, where 1 \leq i \leq M * N,

where

{Num}_{k}

represents the convolution of

F_{k}

and the filter BCDE,

Eq. (2)

BCDE = B * C p q - D * E p q,

where

{Den}_{k}

represents the convolution of

F_{k}

and the filter CE added to the constant

A

that is defined in Sec. 2.1.

Eq. (3)

CE = C p q - E p q .

The filters $C p q$ and $E p q$ are 1-D forms of the filters $C_{i j}$ and $E_{i j}$ as described in Sec. 2.1. The parameters making the filters have suggested the values by Carpenter and Grossberg and can be tuned for optimization of the image evaluation results. The OFF-center IMFs are computed similarly. Double opponent processing fusion is realized by combining pairs of single opponent IMFs obtained from input images as summarized in Eq. (4) and detailed in Fig. 4.

Eq. (4)

[\begin{matrix} Y \\ C b \\ C r \end{matrix}] = {[\begin{matrix} I R_Y^{+}, I R_Y^{-} \\ EO_{Blue}^{+}, EO_{Yellow}^{-} \\ EO_{Red}^{+}, EO_{Green}^{-} \end{matrix}]}^{OnIMFs} .

The sign indicates the center-surround filtering; (+) is for ON and (−) for OFF. Equation (4) summarizes the relationship governing the activities in the retina where the excitation and the inhibition are performed by filtering. The IR image in this work is a white-hot IR image. The design extracts the luminance (grayscale) from the IR image. Its chrominance signals (blue and red) may be added to the chrominance obtained after opponent processing (within and cross modalities) if the IR color information representing cold/warm objects are to be shown. The luminance

Y

of IR is opponent-processed generating an ON center-surround signal (

R_Y^{+}

) and an OFF center-surround signal (

IR_Y^{-}

). These signals carry information about details in the luminance and are averaged forming the fused-image luminance. The color image has its noise removed through a median filter then generates a fourth channel (yellow) by averaging its red and green channels. Following neural activities in the retina, the blue channel IMF is excited (

EO_{Blue}^{+}

) whereas the yellow is inhibited (

EO_{Yellow}^{-}

) to generate the blue chroma IMF. The red channel IMF is excited (

R_Y^{+}

), whereas the green is depressed (

IR_Y^{-}

) to produce the red chroma IMF. The achromatic information has a high spectral sensitivity. IMFs in each channel are summed, generating three 1-D signals (Y, Cb, and Cr) converted into 2-D arrays, the fused image. When merging images of different modalities such as CCDs and IRs, the approach preserves features and edges due to its ability to separate spatial frequencies.

4. Experimental Results

The state-of-the-art techniques for color composite image fusion may be subdivided into opponent processing and improved IHS algorithms. In the field of opponent processing, major works have been done by the teams of Dr. Toet, Dr. Waxman, and Dr. Huang.¹⁴^,¹⁸^,¹⁹ The recent works on color opponent fusing techniques¹⁴ and multispectral image fusion³⁷ justify the choice of the algorithms we chose for performance comparison. In order to evaluate the Bio-EMD fusion, testing was conducted on all the pairs of registered images available in Ref. 38. The performance was consistent throughout the testing samples and we are presenting three of the datasets. For each dataset, our result was compared to results obtained utilizing Toet, Waxman, Huang, and Nunez techniques.

4.1.

Image Fusion

Among the sets of IR and EO images fused, three sets representing different scenes are presented here.

Figure 5 presents the input images and their fusion results. The visible image displays a field view partially obstructed by smoke, the IR image captures thermal differences in areas that are obstructed. Some reference features are the color of the roof in the EO image and the people standing in the IR image. These features are depicted in the fused image. Our model generated enhancement of the fused image. Color regions barely captured in the visible image show details the other techniques did not bring out. Also the new colors are close to reality.

Fig. 5

Dataset 1 (a) EO image, (b) IR image, (c) fusion image employing Toet method, (d) fusion image employing Waxman method, (e) fusion image employing Huang, (f) fusion image employing Nunez method, and (g) fusion image employing Bio-EMD method [(a) and (b) courtesy of Ref. 38].

Figure 6 presents the input images and their fusion results. The reference feature is a group of people in the woods under limited lighting, but depicted in the IR imagery. The fused image rendered the vegetation and the people. This dataset confirms the enhancement properties of our method. The all dark visible image has details bare eyes could not identify but the Bio-EMD filtered out. The dataset however shows the limitations on color enhancement. To have a colorful fused image reflecting reality, the visible image is required to capture some color difference so that opponent processing enhances the information carried by the different IMFs.

Fig. 6

Dataset 2 (a) EO image, (b) IR image, (c) fusion image employing Toet method, (d) fusion image employing Waxman method, (e) fusion image employing Huang, (f) fusion image employing Nunez method, and (g) fusion image employing Bio-EMD method [(a) and (b) cstesy of Ref. 38].

Figure 7 presents the input images and their fusion results. The reference is a crouched down person unseen in Fig. 7(a), EO image, and difficult to depict in Fig. 7(b), IR image. The visible depiction provides no information about the scene, details are not perceptible; the IR image suggests that there is a crouched person on the scene and little can be said about the scene background. Figure 7(g) shows the Bio-EMD fusion result. Figure 7(g) shows the reference feature and the background can be identified, i.e., vegetation. Equation (4) presents the synthesis of Cb channels from blue signal excited and yellow signal inhibited. To get a Cb channel with significant information, some blue and yellow signals need to be present. The same is true with red and green signals to generate a Cr channel. The visible input image visibly lacks these two pairs of color, going through the fusion process. This justifies the low colorfulness of the fused image compared to the other two dataset results and presents the limitations of this algorithm. However, our method delivered a fused image showing sufficient details to detect and recognize objects on the scene. Objective evaluation relies only on the analysis of original images in comparison with fused image results. The evaluation process focuses on preservation of useful information and fused image depiction.

Fig. 7

Dataset 3 (a) EO image, (b) IR image, (c) fusion image employing Toet method, (d) fusion image employing Waxman method, (e) fusion image employing Huang, (f) fusion image employing Nunez method, and (g) fusion image employing Bio-EMD method [(a) and (b) courtesy of Ref. 38].

4.2.

Assessment of Color Fusion Image Quality

The objective evaluation of fused images depends on the amount of information retained from the input images, the edge raggedness, the distinction between bright and dark pixels, and the vividness of the object representation.⁹^,³⁹

4.2.1.

Mutual information

The first evaluation criterion is the well-known mutual information (MI). In this contest, MI evaluates the quantity of information transferred from input images to fused image Z. Piella proposed the MI, $I$ , between two inputs images $X$ and $Y$ fused to generate a composite image $Z$ as the sum of the MI between the composite image and each of the inputs, divided by the sum of the entropies of the input images⁴⁰

Eq. (5)

I (X, Y, Z) = \frac{I (X, Z) + I (Y, Z)}{H (X) + H (Y)} .

I (X, Z)

is the MI between an image candidate to fusion and the resulting image;

H (X)

is the entropy of the image

X

. The higher the MI between

X

,

Y

, and

Z

, the more the information is transferred to

Z

. Thus, MI is a similarity measure. Table 3 contains the MI for the three datasets. The metric is computed for a fused image generated by each of the techniques tested.

Table 3

Mutual information results.

	Dataset 1	Dataset 2	Dataset 3
Toet method	0.0981	0.0714	0.0745
Waxman method	0.2685	0.0634	0.0629
Huang method	0.1164	0.0558	0.0795
Nunez method	0.0958	0.0523	0.0795
Bio-EMD method	0.3738	0.1722	0.2322

4.2.2.

Sharpness

The second metric utilized is the image sharpness metric (ISM) developed by Yuan and her colleagues¹⁰ and defined as

Eq. (6)

ISM = \frac{1}{| W |} \sum {(G_{x}^{2} + G_{y}^{2})}^{\frac{1}{2}} .

| W |

is the total number of

w

(

3 \times 3

size windows) and

G_{x}

and

G_{y}

represent the Sobel operator at a pixel (

x

,

y

). Color image quality attribute sharpness is related to the clarity of details and definition of edges. Sharpness of an image includes details, line quality, adjacency, effective resolution, edge sharpness, and edge raggedness.³⁹^,⁴¹ Sharpness can be measured by the edge information. With a color image, sharpness relates to its luminance and therefore the gray intensity of the image. Table 4 presents sharpness evaluation of our technique and some others.

Table 4

Sharpness results.

	Dataset 1	Dataset 2	Dataset 3
Toet method	12.2139	4.9326	5.9994
Waxman method	6.9070	7.0146	8.4139
Huang method	11.4983	9.5377	8.4139
Nunez method	5.9944	13.5585	14.1227
Bio-EMD method	15.0809	23.99646	20.7252

4.3.

Contrast

Contrast is the perceived magnitude of visually meaningful differences, global and local, in lightness and chromaticity within the image.³² Contrast of an image is a perceptual attribute representing the ratio between the brightest pixel and the darkest pixel intensities. This is a dynamic range where higher values indicates better image contrast and lower values are synonym of lower contrast, lower quality. Many metrics have been developed for contrast evaluation in grayscale images.²⁹^,⁴¹ Yuan and her colleagues proposed to employ the $L^{*}$ channel from Commission Internationale de l’Eclairage standard CIE 1976 $L^{*} a^{*} b^{*}$ color space to evaluate the color contrast since human perception is more sensitive to the luminance in contrast evaluation. Equation (7) defines their proposed image contrast metric (ICM)⁹

Eq. (7)

ICM = {(w_{1} \times C_{g}^{2} + w_{2} \times C_{c}^{2})}^{\frac{1}{2}},

where

C_{g}

and

C_{c}

represent the gray contrast and color contrast metric,

w_{1}

and

w_{2}

, and their corresponding weights;

w_{1} = w_{2} = 0.5

⁹.

Eq. (8)

C_{g} = α_{I} \sum_{k = 0}^{N_{I} - 1} \frac{I_{k}}{N_{I}} P (I_{k}) C_{c} = α_{L^{*}} \sum_{k = 0}^{N_{L} - 1} \frac{L_{k}^{*}}{N_{I}} P (L_{k}^{*}),

where

P (I_{k})

and

P (L_{k}^{*})

are the probability density functions of the gray intensity

I

and the CIELAB

L^{*}

N_{I}

and

N_{L^{*}}

are the total number of levels.

I

ranges from 0 to 255 while

L^{*}

ranges from 0 to 100.

α_{I}

and

α_{L^{*}}

represent the dynamic ranges for intensity and color such that

Eq. (9)

α = \frac{N_{1}}{N + N_{2}},

where

N

indicates the number of pixel levels,

N_{1}

the number of pixel levels with nonzero count, and

N_{2} = N - N_{1}

. Table 5 presents contrast evaluation results in fused images generated by of our method and some other techniques.

Table 5

Contrast results.

	Dataset 1	Dataset 2	Dataset 3
Toet method	73.0746	44.8277	117.2229
Waxman method	126.2539	46.4329	69.7677
Huang method	36.0826	39.431	69.7677
Nunez method	4.2732	3.6351	44.8418
Bio-EMD method	29.4697	18.0539	17.4365

4.4.

Colorfulness

Color depiction and rendering is one of the major differences between current and past imaging systems. Colorfulness, also referred to as “chromaticness,” is the attribute of a visual sensation according to which the perceived color of an area appears to be more or less chromatic.³⁰^,³¹ Yuan and her colleagues proposed a different approach based on color chroma metric ${CCM}_{1}$ and color variety metric ${CCM}_{2}$ such that the image colorfulness metric CCM is defined as

Eq. (10)

CCM = \frac{1}{2} \times ({CCM}_{1} + {CCM}_{2}),

where the chroma metric is defined by Eq. (11) and the variety metric is computed as presented in Eq. (12).⁹

Eq. (11)

{CCM}_{1} = \sqrt{\frac{1}{M \times N} \sum_{i = 0}^{N} \sum_{j = 0}^{M} C_{i j}^{*}},

where

C^{*}

represents the component computed in Eq. (12).²⁷

Eq. (12)

C^{*} = {(a^{* 2} + b^{* 2})}^{\frac{1}{2}} h^{*} = \arctan (\frac{b^{*}}{a^{*}}),

Eq. (13)

{CCM}_{2} = \sqrt{\frac{1}{| w |} \sum \nabla f (i, j | w)},

where

| W |

is the total number of all windows (

w

); the color difference gradient of pixel

f (x, y)

is defined in Ref. 9. Colorfulness metric proposed by Hasler and Susstrunk³¹ generated results similar to the ones in Table 6.

Table 6

Colorfulness results.

	Dataset 1	Dataset 2	Dataset 3
Toet method	0.8619	0.7735	0.8333
Waxman method	0.7887	0.7481	0.7355
Huang method	0.8675	0.7481	0.7355
Nunez method	0.9451	0.7006	0.6165
Bio-EMD method	0.7279	0.6963	0.6233

Bio-EMD transferred twice as much information as each of the others. It also generated twice the sharpness of the other techniques, and displayed more natural looking colors. Bio-EMD generated contrast values in the same range as other approaches. Colorfulness was weak based on the metric although the depiction is meaningful compared to others.

In general, each of these three datasets projected the same performance with the Bio-EMD method. Bio-EMD technique transferred the information from input images to output image better than the other techniques which was compared with MI. Also, Bio-EMD provided a better clarity of details and definition of edges (ISM). Although Bio-EMD resulted in more natural looking colors in the output image, the image vividness metric (ICC) utilized and others tested during this work did not always convey that strength. The evaluation of our result perceived magnitude of visually meaningful differences, global and local, in lightness and chromaticity, is in close range with the other methods (ICM). Better quantitative results may be obtained by tuning the filter parameters.

5. Conclusion

The analysis of existing fusion techniques applied to visible and IR images showed a need for an approach that performs color fusion of these two modalities and generates high-quality images with near true color. This work presents the development and testing of a new image fusion method based on EMD and opponent processing. EMD represents input images as IMFs carrying their spatial and frequency components about each pixel. Following a human retinal model, IMFs from visible and IR sources are combined utilizing the proposed network of dynamic equations feeding YCbCr channels of the output section. Testing was done on all the registered pairs of images available in Ref. 38 and the performance was consistent throughout all the samples.

Observation of the resulting images shows significant improvement compared to previously developed procedures. Quantitative assessment of the fused image attributes consisted of four metrics: MI, sharpness, colorfulness, and contrast. These metrics confirmed that the proposed approach generated twice the information transfer from original images compared to existing techniques. The clarity of details was comparable to the major color fusion techniques. Contrast generated in the fused images was adequate; however vividness of the images was subpar although the fused images obtained had more meaningful colors. This highlights the lack of a fused image metric.

The Bio-EMD algorithm produced imagery with higher information content than either the low-light visible or IR input image taken separately. Fusion generated a depiction of objects seen only in one modality or not seen in the original images. This breakthrough can be applied in multisensor fusion involving visible bands and has applications in situational awareness, forensics, medical imaging, remote sensing, natural resources management, etc. The breaking point of this method is that it performs well only when there is a minimum of color information in the visible image; a visible image with no or very low-color information will produce a fused image with low quality. This is echoed by dataset 3 in Fig. 7. To get a Cb channel in fused image with significant information, some blue and yellow signals need to be present. The same is true with red and green signals to generate a Cr channel. The visible input image visibly lacks these two pairs of color, going through the fusion process. This justifies the low colorfulness of the fused image compared to the other two dataset results and presents the limitations of this algorithm. In order to obtain the best result, this fusion algorithm requires some minimum information from the pair of signals blue–yellow and red–green. How much information do we need? What is the threshold of color level in visible input image? The answers to these questions are the object of on-going research where we also consider the different parameters of the enhancement/inhibition filters. This will define the conditions of its applications.

Acknowledgments

The authors express their gratitude to Dr. Nicholas Beser, Dr. Richard Dean, and Dr. Alexander Toet for their feedback, Dr. Starvi Nikolov and Yihui Yuan for providing us with images, and the School of Electrical and Computer Engineering at Morgan State University. The authors would like to thank the reviewers and editors for their very helpful comments. The research was supported by Chesapeake Information Based Aeronautical Consortium (CIBAC).

References

1.

T. Stathaki, Image Fusion: Algorithms and Applications, Academic Press, London (2008). Google Scholar

2.

G. CarpenterS. Grossberg, Neural Networks for Vision and Image Processing, MIT Press, London (1992). Google Scholar

3.

G. Piella, “A general framework for multiresolution image fusion: from pixels to regions,” Inform. Fusion, 4 (4), 259 –280 (2003). http://dx.doi.org/10.1016/S1566-2535(03)00046-0 1566-2535 Google Scholar

4.

M. XuH. ChenP. Varshney, “An image fusion approach based on Markov random fields,” IEEE Trans., 49 (12), 5116 –5127 (2011). http://dx.doi.org/10.1109/TGRS.2011.2158607 IGRSD2 0196-2892 Google Scholar

5.

A. Masiniet al., “Sight enhancement through video fusion in a surveillance system,” in IEEE 14th Int. Conf. Image Analysis and Processing, (2007). Google Scholar

6.

M. HolschneiderP. Tchamitchian, Les ondelettes, 102 Springer-Verlag, Berlin (1990). Google Scholar

7.

A. ToetJ. Walraven, “New false color mapping for image fusion,” Opt. Eng., 51 (1), 650 –658 (1996). http://dx.doi.org/10.1117/1.600657 OPEGAR 0091-3286 Google Scholar

8.

B. YangF. SunS. Li, “Region-based color fusion method for visible and IR image sequences,” in Chinese Conference on Pattern Recognition, 1 –6 (2008). Google Scholar

9.

Y. Yuanet al., “Objective quality evaluation of visible and infrared color fusion image,” Opt. Eng., 50 (3), 033202 (2011). http://dx.doi.org/10.1117/1.3549928 OPEGAR 0091-3286 Google Scholar

10.

N. Huanget al., “The empirical mode decomposition and Hilbert spectrum for non-linear and non-stationary time series analysis,” Proc. Roy. Soc. A, 454 903 –995 (1998). http://dx.doi.org/10.1098/rspa.1998.0193 PRLAAZ 0080-4630 Google Scholar

11.

E. KandelJ. SchwartsT. Jessell, Principles of Neural Science, 4th ed.McGraw-Hill, Oxford (2000). Google Scholar

12.

P. SchillerJ. SandellJ. Maunsell, “Functions of the ON and OFF channels of the visual system,” Nature, 322 (6082), 824 –825 (1986). http://dx.doi.org/10.1038/322824a0 NATUAS 0028-0836 Google Scholar

13.

P. Schiller, “The ON and OFF channels of the visual system,” Trends Neurosci., 15 (3), 86 –92 (1992). http://dx.doi.org/10.1016/0166-2236(92)90017-3 TNSCDR 0166-2236 Google Scholar

14.

A. ToetM. Hogervorst, “Progress in color night vision,” Opt. Eng., 51 (1), 010901 (2012). http://dx.doi.org/10.1117/1.OE.51.1.010901 OPEGAR 0091-3286 Google Scholar

15.

A. Waxmanet al., “Opponent color fusion of multi-sensor imagery: visible IR and SAR,” in Meeting of the IRIS Special Group on Passive Sensors I, (1998). Google Scholar

16.

A. Waxmanet al., “Color night vision; opponent processing in the fusion of visible and IR imagery,” Neural Netw., 10 (1), 1 –6 (1997). NNETEB 0893-6080 Google Scholar

17.

A. Waxmanet al., “Solid state color night vision: fusion of low-light visible and thermal IR imagery,” (1996). Google Scholar

18.

A. Fayet al., “Fusion of multi-sensor imagery for night vision: color visualization, target learning and search,” in Third International Conference on Image Fusion, (2000). Google Scholar

19.

G. HuangG. NiB. Zhang, “Visual and infrared dual-band false color image fusion method motivated by Land’s experiment,” Opt. Eng., 46 (2), 027001 (2007). http://dx.doi.org/10.1117/1.2709851 OPEGAR 0091-3286 Google Scholar

20.

F. Sadjadi, “Comparative image fusion analysis,” in IEEE Conf. Computer Vision and Pattern Recognition, 8 (2005). Google Scholar

21.

J. Nunezet al., “Multiresolution-based image fusion with additive wavelet decomposition,” IEEE Trans. Geosci. Rem. Sens., 37 (3), 1204 –1211 (1999). http://dx.doi.org/10.1109/36.763274 IGRSD2 0196-2892 Google Scholar

22.

P. MassipP. BlancL. Wald, “A method to better account for modulation transfer functions in ARSIS-based pansharpening methods,” IEEE Geosci. Rem. Sens., 50 (3), 800 –808 (2012). http://dx.doi.org/10.1109/TGRS.2011.2162244 IGRSBY 1545-598X Google Scholar

23.

X. Otazuet al., “Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods,” IEEE Geosci. Rem. Sens. Lett., 43 (10), 2376 –2385 (2005). http://dx.doi.org/10.1109/TGRS.2005.856106 IGRSBY 1545-598X Google Scholar

24.

G. Piella, “Adaptive wavelets and their applications to image fusion and compression,” Lehigh University, (2003). Google Scholar

25.

T. PappasR. Safranek, “Perceptual criteria for image quality evaluation,” Handbook of Image and Video Processing, Academic Press, New York (2000). Google Scholar

26.

V. Zujovicet al., “A new subjective procedure for evaluation and development of texture similarity metrics,” in 10th IVMSP Workshop, 123 –128 (2011). Google Scholar

27.

V. TsagarisV. Anastassopoulos, “Global measure for assessing image fusion methods,” Opt. Eng., 45 (2), 026201 (2006). http://dx.doi.org/10.1117/1.2174005 OPEGAR 0091-3286 Google Scholar

28.

M. Pedersenet al., “Attributes of image quality for color prints,” J. Electron. Imag., 19 (1), 011016 (2010). http://dx.doi.org/10.1117/1.3277145 JEIME5 1017-9909 Google Scholar

29.

R. FrazorW. Geisler, “Local luminance and contrast in natural images,” Vis. Res., 46 (10), 1585 –1598 (2006). VISRAM 0042-6989 Google Scholar

30.

H. Palus, “Colorfulness of the image: definition, computation and properties,” Proc. SPIE, 6158 615805 (2006). http://dx.doi.org/10.1117/12.675760 Google Scholar

31.

D. HaslerS. Suesstrunk, “Measuring colorfulness in natural images,” Proc. SPIE, 5007 87 –95 (2003). http://dx.doi.org/10.1117/12.477378 Google Scholar

32.

K. ZhangY. LiuJ. Chen, “Fusion of the infrared and color visible images using bidimensional EMD,” in IEE Int. Conf. Multimedia and Information Technology, 257 –260 (2009). Google Scholar

33.

G. Rillinget al., “Bivariate empirical mode decomposition,” IEEE Trans. Signal Process., 14 (12), 936 –939 (2007). http://dx.doi.org/10.1109/LSP.2007.904710 ITPRED 1053-587X Google Scholar

34.

D. LooneyP. Mandic, “Multiscale image fusion using complex extensions of EMD,” IEEE Trans., 57 (4), 1626 –1630 (2009). http://dx.doi.org/10.1109/TSP.2008.2011836 ITPRED 1053-587X Google Scholar

35.

X. XiaohongN. Jiping, “Review of EMD based image fusion,” in Int. Conf. Intelligence and Information Engineering, 282 –285 (2011). Google Scholar

36.

B. Altafet al., “Rotation invariant complex empirical mode decomposition,” in IEEE Int. Conf. Acoustics, Speech and Signal Processing, 1009 –1012 (2007). Google Scholar

37.

X. Otazuet al., “Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods,” IEEE Trans. Geosci. Rem. Sens., 43 (10), 2376 –2385 (2005). http://dx.doi.org/10.1109/TGRS.2005.856106 IGRSD2 0196-2892 Google Scholar

38.

O. Ltd., (2011) http://www.imagefusion.org January ). 2011). Google Scholar

39.

M. TseA. Klein, “Automated print quality analysis in inkjet printing: case study using commercially available media,” in Int. Conf. Digital Printing Technologies, 176 –171 (1998). Google Scholar

40.

G. Piella, “Adaptive wavelets and their applications to image fusion and compression,” Lehigh University, (2003). Google Scholar

41.

E. Peli, “Contrast in complex images,” J. Opt. Soc. Am., 7 (10), 2032 –2040 (1990). http://dx.doi.org/10.1364/JOSAA.7.002032 JOSAAH 0030-3941 Google Scholar

Biography

Paterne Sissinto received his doctorate in electrical engineering from Morgan State University in 2012, where he worked on pattern recognition and image processing. He is currently teaching at Prince George’s Community College.

Jumoke Ladeji-Osias is an associate professor and associate chair for graduate studies in the Department of Electrical and Computer Engineering at Morgan State University in Baltimore, Maryland. She earned a BS in electrical engineering from University of Maryland, College Park, in 1993, and a PhD in biomedical engineering from Rutgers, The State University of New Jersey, in 2000. Her research interests include FPGA based digital hardware implementations and engineering education.

CC BY: © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation Download Citation

Paterne S. Sissinto and Jumoke O. Ladeji-Osias "Bio-empirical mode decomposition: visible and infrared fusion using biologically inspired empirical mode decomposition," Optical Engineering 52(7), 073101 (8 July 2013). https://doi.org/10.1117/1.OE.52.7.073101

Published: 8 July 2013

Access the abstract

JOURNAL ARTICLE
12 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 2 scholarly publications.

Explore citations on Lens.org

KEYWORDS

Image fusion

Infrared imaging

Image processing

Visible radiation

Infrared radiation

RGB color model

Optical filters

1.

Introduction

2.

Background and Concepts

2.1.

Dynamic Neural Network Equations

Table 1

Table 2

2.2.

Color Fusion of IR and Visible Images

Fig. 1

2.3.

Evaluation of Color Fusion

2.4.

Empirical Mode Decomposition

Fig. 2

3.

Bio-EMD Fusion of IR and Visible Images

Fig. 3

Fig. 4

Eq. (1)

Eq. (2)

Eq. (3)

Eq. (4)

4.

Experimental Results

4.1.

Image Fusion

Fig. 5

Fig. 6

Fig. 7

4.2.

Assessment of Color Fusion Image Quality

4.2.1.

Mutual information

Eq. (5)

Table 3

4.2.2.

Sharpness

Eq. (6)

Table 4

4.3.

Contrast

Eq. (7)

Eq. (8)

Eq. (9)

Table 5

4.4.

Colorfulness

Eq. (10)

Eq. (11)

Eq. (12)

Eq. (13)

Table 6

5.

Conclusion

Acknowledgments

References

Biography

Show All Keywords

Keywords/Phrases

Search In:

Publication Years