The frequency range of a millimeter wave (MMW) is 30 to 300 GHz.1 The frequency range of terahertz (THz) is normally 100 GHz to 10 THz.2 The wavelength of MMW is longer; it has higher atmospheric transmittance, and in the process of propagation, poor environmental conditions have less impact on it.1 The frequency of THz is higher; it has higher spatial resolution or longer depth of field when the spatial resolution is consistent.2 Therefore, the images of the two bands may contain different information. Since the human body radiates MMW and THz waves, the two bands are often used for human body security imaging. Compared with commonly used MMW or THz single-band imaging systems, MMW and THz dual-band imaging systems can acquire more information because the images obtained using image fusion algorithms contain more accurate descriptions of the scene than single-band images could.34.–5
In our investigation, we employed an MMW and THz dual-band passive human-body security imaging system composed of 94 and 250 GHz single-element detectors to study MMW and THz dual-band image preprocessing and fusion algorithms.6,7 We developed an algorithm called the MMW and THz image preprocessing and fusion integrated algorithm (MMW-THz IPFIA), which is proposed herein. The MMW-THz IPFIA consists of denoising, enhancement, registration, and fusion algorithms and yields fused images that contain most of the information from the corresponding dual-band images, improves the probability of detection, and enhances the performance of the dual-band imaging system. The effectiveness of the algorithm is analyzed by using image signal-to-noise ratio (SNR) and information entropy.89.–10 The performance improvement provided by the MMW-THz IPFIA to the system is also demonstrated in this report. Because the system responds to blackbody radiation, the commonly used thermal imaging system performance evaluation parameter—the minimum detectable temperature difference (MDTD)—was employed to analyze the system’s performance. Finally, we performed an experiment in which the MDTD was measured both before and after applying the MMW-THz IPFIA to obtain objective, quantitative evidence of the improvements it yields.
Millimeter Wave and Terahertz Dual-Band Passive Human-Body Security Imaging System
Figure 1 depicts a schematic diagram of a MMW and THz dual-band passive human-body security imaging system, which is composed of a plane mirror, trihedral focusing scanning mirror, reflector, beam splitter, 94-GHz single-element detector, and 250-GHz single-element detector. The plane mirror collects the MMW and THz radiation emitted by the human body as it is swept along the direction perpendicular to the image plane to conduct a line scan. The trihedral focusing scanning mirror focuses the target radiation on the reflector and rotates along the direction parallel to the image plane to perform a column scan. The reflector reflects the radiation to the beam splitter. The beam splitter is an optical element with a semitransparent, semireflective characteristic, and it can divide the radiation into two beams: the transmitted beam and the reflected beam. The transmitted beam and the reflected beam can be captured by the 94- and 250-GHz detectors, respectively, and used to form image signals and create dual-band images.
As shown in Fig. 2(a), the images obtained by the 94 and 250 GHz detectors depict a test subject hiding a steel ruler in front of the abdomen. Figures 2(b) and 2(c) present similarly generated images of the subject with a THz attenuator in front of the left side of the chest and with papers hidden within the subject’s clothes, respectively. In these original images, there is significant noise, the resolution and contrast are low, and affine problems are obvious. Thus, it is necessary to preprocess such images by performing denoising, enhancement, and registration before fusing them to obtain high SNRs, high contrast, and rich details in the resulting fused images.
Millimeter Wave and Terahertz Image Preprocessing and Fusion Integrated Algorithm
In this report, we propose an MMW-THz IPFIA, which can be used to preprocess and fuse MMW and THz dual-band human-body images.
Millimeter Wave-Terahertz Image Preprocessing and Fusion Integrated Algorithm
The MMW-THz IPFIA employs a block matching and three-dimensional (BM3D) filtering denoising algorithm, a contrast-limited adaptive histogram-equalization (CLAHE) enhancement algorithm, a mutual information based registration algorithm, and a wavelet transform-based image-fusion algorithm.
Block matching and three-dimension
The BM3D algorithm, a three-dimensional (3-D) transform-domain filtering algorithm based on block matching, is one of the most effective image denoising algorithms currently in use. Figure 3 presents the structure of the BM3D algorithm, which consists of two steps: basic estimation and final denoising.11,12
Basic estimation: Obtaining a basic estimate requires three steps.
1. Block matching and grouping: By sliding a window with dimensions of matching a certain step length, divide Image I into several blocks. Assume that is the currently selected block; is the search diameter; is the sliding block in the search region; and and , which represent the locations of the blocks, are the values of the pixels in the top left corners or the centers of blocks and , respectively. Then the distance between the two blocks is . If an appropriate distance threshold is selected, where , then is the similarity block of . In addition, the set of the similarity blocks of forms a 3-D matrix , where .
2. 3-D Transform Domain Denoising: First, arrange the elements of in order of the size of . Next, perform a one-dimensional (1-D) Haar wavelet transfer between the blocks and a two-dimensional (2-D) Bior wavelet hard-threshold filtering within the block. Then the transform coefficient of the hard threshold filtering is , which can be modulated by
After performing that operation, write the denoised information back to the original position in the image by conducting an interblock inverse transformation. The 3-D transform denoising equation is as follows:
In Eq. (1), is the numeric of in the matrix, is the hard threshold shrinkage parameter, is the estimated standard deviation of the noise, and is the denoised 3-D image of the block matrix.
3. Aggregation: After step (2) has been completed, an estimate will have been performed for every block and element in the image, and the similarity block set of , , will have changed to . In , is the number of nonzero coefficient blocks, is the basic estimated weight of , and the equation used to calculate is as follows:
Because the step length of the sliding window may be less than its size, the blocks may overlap. Thus, an element may appear in different blocks. Estimating the value of for the ’th element requires reassessing the overlapping blocks by using the weighted average as follows:
The final denoising process involves three steps.
1. Block matching and grouping: Match and group the denoised images obtained from the basic estimate, generating a new 3-D matrix .
2. Joint Wiener filtering denoising: Transform and by applying a 3-D transformation (2-D DCT cosine transform and 1-D Haar wavelet transform). Use to perform Wiener filtering of to obtain the final weight estimate . Write the estimator of every pixel back to the original position by applying the interblock inverse transformation:
3. Aggregation: Re-estimate the overlapped blocks and obtain the result by calculating the weighted average:
Image enhancement technology can improve the visibility of barely observable targets and enhance the level of detail of the information acquired. Some small targets and detailed texture features occupy relatively few pixels in MMW and THz images. Thus, they occupy smaller portions of the images’ grayscales. In the proposed method, weak target loss during the fusion process is avoided since a CLAHE algorithm is used to enhance the denoised images.13,14
The CLAHE algorithm, a classic image enhancement algorithm, involves four steps.
1. Blocking: Use a sliding window to pass over the image. The grayscale of the image is . Acquire a series of local image sub-blocks with dimensions of .
2. Calculate the probability of grayscale , , and obtain the normalized histogram and probability distribution function (PDF) of every sub-block. In the preceding equation, represents the ’th grayscale, is the number of pixels in grayscale , is the number of pixels in the sub-block, and , where is the gray level.
3. As depicted in Fig. 4, for the PDF of each sub-block, set a threshold: . The part of the PDF that is larger than is the intercepted portion, the lower and upper limits of the grayscale of the intercepted portion are and , respectively. If the total number of the gray level in the sub-block is , when the grayscale is , the PDF corresponding to is divided into copies, that is, , then is added to the PDF corresponding to each grayscale in the sub-block. The PDF below is increased, and the PDF above is reduced.
4. Transform the contrast-limited histogram of each sub-block, map the elements of the original grayscale onto grayscale , where
The MMW and THz dual-band, passive human-body security imaging system employed in this investigation uses two different detectors to capture dual-band images. Variations in the detector parameters lead to geometric deviation problems, such as translation, scaling, and rotation between images. It is necessary to correct such geometric deviations to register images. In this study, we employed a mutual-information-based method to register the images, since otherwise the grayscale details in the dual-band images would be minimal and the characteristics of the images would not be obvious.15,16
The mutual-information-based registration method utilizes mutual information to conduct similarity measurements and image registration, and includes the following four basic components:
1. A feature space consisting of the gray values of the images;
2. A search space for global translation, rotation, and scaling;
3. Similarity measurements of mutual information: Mutual information is used to assess the correlation between the information provided by two systems. In the system employed in this study, although the two images in each set were obtained by different detectors, they were based on the same scene information. Thus, when the spatial positions of the two images in a set are identical, the value of the information that one image provides about the other, that is, the mutual information, was maximized. Algorithms typically use the generalized distance between the joint probability distribution and separate probability distributions to estimate the mutual information. Here, this relation is as follows:
For images and , the corresponding pixel gray values are and , respectively, which are linked by a coordinate transformation. The joint distribution can be calculated using the normalized joint grayscale histogram . Thus,
The marginal probability distributions are and .
4. A search strategy employing the (1 + 1) evolution strategy algorithm [(1 + 1) ES]: This algorithm is a numerical optimization algorithm for solving continuous search spaces that was proposed in the 1960s by German mathematicians, Rechenberg and Schwefel, and occupies an important position in research dedicated to and applications of evolution strategy.17,18
Each generation in (1 + 1), ES consists of only two individuals: a parent and an offspring. The offspring is generated through a mutation of the parent; the better of the parent and offspring is chosen as the next generation’s parent. This process can be summarized by the following four steps:
Image fusion involves integrating information from two or more source images to obtain a new image that can present the scene exactly, entirely, and reliably. Compared with the prefusion images, a fused image contains more information, more details, and greater clarity. Thus, the potential target detection rate is greater. The wavelet-transform-based fusion method takes full advantage of the wavelet’s multiresolution analysis capabilities by dividing the original image into a series of subimages with different spatial resolutions and frequency domain features. The subimages reflect the local characteristics of the original image. In this method, the subimages are fused according to certain rules, the multiscale description of the fused image is formed, and, finally, the fused image is obtained by performing inverse-wavelet-transform reconstruction. Figure 5 presents a schematic diagram of the steps of the fusion algorithm.19,20
The wavelet base used in this research is the simplest Haar wavelet base:
The original image is decomposed by performing a three-level discrete wavelet transform. The image characteristics are acquired using different resolutions, and after Eq. (13) is employed to complete the computation of the weighted average of the wavelet coefficient, the fused image is obtained:
Algorithm Evaluation and Verification
Objective evaluation parameters
We use SNR (SNR is a measure of the strength relationship between the signal and the noise) and the information entropy (the larger the value, the richer the details contained in the image) to analyze the performance of the algorithm.
For an image , it is first divided into several sub-blocks by sliding a window with dimensions of on it. Then the local mean and the local variance of the sub-bocks are calculated.9 Suppose that the central pixel of a block is , the local mean and the local variance are as follows:
The maximum value of the local variance is selected as the signal variance of the image:
The SNR of the image is as follows:
For an image I, the information entropy is as follows:10
In this section, indicators of the effectiveness of applying the MMW-THz IPFIA to the images depicted in Fig. 2 are discussed. Figures 6Fig. 7–8 correspond to Figs. 2(a)–2(c), respectively. From left to right in Figs. 6–8, the profiles are presented in three groups. The first and second groups of four images, which were acquired using the 94 and 250 GHz detectors, respectively, each include the original, denoised, enhanced, and registered images, from left to right. The image on the far right in each figure is the final fused image.
The processed images display reduced noise, improved detail contrast, and effective affine problem correction. The features of the two-band image pairs are synthesized in the fused images, which contain more detail. The probability of detection and capabilities of the dual-band passive THz human-body security imaging system are improved by applying the MMW-THz IPFIA.
According to Eqs. (14)–(18), the SNR of the original image and the fused image are calculated first. The size of the sliding window is . The rows of the flat area are [10 25] and the columns are [30 50] (as shown in Figs. 6–8, the area in the red lines is the flat area). The grayscale distribution of the selected area in the fused images is flat. It does not contain edges or significant changes. Moreover, in this area, there is only noise in the original images, and the grayscale distribution is also flat.
Table 1 shows that the values of SNR of the fused image are several times larger than that of the original image. This indicates that the noise in the image is obviously reduced.
SNRs of the original image and the fused image.
|Fig. 2(a) steel ruler||Fig. 2(b) THz attenuator||Fig. 2(c) original papers|
|94 GHz||250 GHz||94 GHz||250 GHz||94 GHz||250 GHz|
Table 2 shows the values of information entropy of the original image and the fused image. The values are increased by 1 basically (for 8-bit digital images, the maximum of the information entropy is 8). This indicates that the detail information contained in the image is enhanced effectively.
Information entropy of the original image and the fused image.
|Fig. 2(a) metal ruler||Fig. 2(b) beam splitter||Fig. 2(c) printing paper|
|94 GHz||250 GHz||94 GHz||250 GHz||94 GHz||250 GHz|
The two parameters fully illustrate that the algorithm can effectively reduce the noise and improve the level of detail in the images. The fused image contains more details than the original image, thus it can describe the scene more precisely.
Minimum Detectable Temperature Difference Measurement and Verification
MDTD is an important parameter for evaluating the performances of thermal imaging systems. It can reflect a system's thermal sensitivity characteristics and spatial resolutions. Furthermore, the MDTD does not restrict the duration of the observation. The MDTD can be measured as soon as the position of a square or circular blackbody target in the image displayed on the screen starts to become identifiable, as illustrated in Fig. 9, and equals the temperature difference between the blackbody and background.
The imaging target of the MMW-THz dual band passive human-body security imaging system is the scene at room temperature, therefore, the evaluation of the system performance can be objectively carried out by using the measurement method similar to the thermal imaging system MDTD.
The components of the experimental system designed to measure the MDTD are depicted in Fig. 10. Given that the environmental temperature was 20°C, both the 94 and 250 GHz subsystem unit detectors were 10 mm in size. The focal length of the system was 300 mm; the aperture diameter was 200 mm; the imaging object distance was 1.7 m; the imaging range of the object was ; and the sample spacing was 10 mm. By calculating the optical diffraction limit, the minimum resolvable sizes of the optical system of the 250 GHz subsystem and the 94-GHz subsystem are 2.196 and 5.84 mm, respectively. They are smaller than the size of the detector, therefore, the spatial resolution of the imaging system is mainly affected by the size of the detector.
Since the imaging object distance was 1.7 m, and the MDTD measurement targets were designed to be circular and to range in diameter from 60 to 130 mm in 10-mm increments, the spatial angular frequency is presented in Table 3. Images of the 60- and 100-mm-diameter targets are illustrated in Fig. 11 as examples. The measured MDTD curve is shown in Fig. 12.
Spatial angular frequencies of the targets f (cyc/mrad).
The curve shown in Fig. 12 can be used to compare the original MDTDs with those after preprocessing with the 94 GHz and 250 GHz subsystems. In principle, for the same spatial angular frequency, the smaller the MDTD, the higher the system detection rate. In each case, the MDTD is reduced by preprocessing, indicating that the detection performance of the system is improved. Under identical conditions, the MDTD of the 94-GHz subsystem is larger than that of the 250-GHz subsystem in both the original and processed systems. Thus, the MDTD of the system processed by using MMW-THz IPFIA lies between those of the processed 94- and 250-GHz subsystems, that is, . In addition, the MDTDs obtained using the proposed algorithm are smaller and are favorable compared with those obtained using the two original subsystems. This outcome, attributable to the difference between the performances of the two subsystems, demonstrates that the final MDTD is related to the proportions of the subsystem images in the fused images. These results quantitatively demonstrate that the detection performance of the system is improved by applying the MMW-THz IPFIA.
In this article, a comprehensive algorithm, MMW-THz IPFIA, was proposed for use in MMW and THz dual-band passive human-body security imaging systems. This algorithm, which consists of denoising, enhancement, registration, and fusion algorithms, can be employed to generate fused images while preserving most of the details present in the original dual-band images. The performance of the algorithm was analyzed by calculating the SNR and information entropy of the original images and fused images. The result indicated that the resulting fused images exhibited less noise, greater detail, and more information than the single-band images did.
We used a test method similar to the MDTD of the thermal imaging system to objectively verify the performance improvement of the system provided by applying the MMW-THz IPFIA. The result indicated that the MDTD performance has been improved. Thus, the performance of the system can be improved by employing the MMW-THz IPFIA.
Since the denoising, enhancement, registration, and fusion algorithms for MMW and THz images cannot yet be employed in real-time processing, the objectives of the next phase of research will be to integrate these processes with hardware systems and to achieve real-time fusion of dual-band images to support efficient security monitoring using MMW and THz dual-band, passive human-body security imaging systems.
This work is supported by the project of National Science Foundation of China (Grant No. 61171051).
Li Tian is a doctoral student at Beijing Institute of Technology in China. Her current research interests include THz imaging theory, THz imaging systems, and THz image processing.
Weiqi Jin is a professor at Beijing Institute of Technology in China. His current research interests include night vision and infrared imaging technology, photoelectric image processing, photoelectric detection and instrumentation.