The advent of modern hyperspectral imaging modalities opened the possibility of implementing a large spectrum of applications that cannot be robustly solved using the standard trichromatic [red, green, blue (RGB)] data. Among many possible applications that were well served by the inclusion of hyperspectral imaging systems (such as remote sensing, biomedical imaging, industrial inspection, etc.),12.3.–4 the accurate recognition of nonferrous materials represents a prominent example.126.96.36.199.10.11.–12 The motivation behind the decision to use hyperspectral information in the nonferrous material sorting process is multifold, where the chief reason being the fact that the color information is not sufficiently descriptive to robustly sample the characteristics associated with each nonferrous fraction. This issue proved to be particularly apparent in our study as the different fractions of nonferrous materials show considerable intraclass dispersion and a relative high interclass similarity in the RGB domain.
As opposed to the standard RGB image-formation process, hyperspectral imaging involves the acquisition and interpretation of multidimensional digital images that are able to sample the properties of the imaged materials in a substantially wider spectral domain than that covered by the visible spectrum.12,13 In this regard, it is worth mentioning that the current range of hyperspectral sensors are able to simultaneously capture hundreds of spectral bands from ultraviolet to far infrared with a spectral resolution of less than 10 nm.14,15 Thus, the main characteristic of the hyperspectral images is that each pixel is defined by a large vector whose elements are the spectral components captured from the light arriving at the spectral sensor.4,13,1617.18.–19 While there is no doubt that the availability of hyperspectral data is opportune in the classification process, it is important to note that the process of capturing a large number of spectral bands results in a high dimensional data that may raise substantial challenges from practical and computational perspectives. In addition to the extensive dimensionality of the hyperspectral data, the substantial correlation between the closely spaced spectral bands may represent another issue that may affect the accuracy of the material classification process. Thus it is clear that the inclusion of the unprocessed hyperspectral data in the classification process is not only suboptimal, but also this approach will compromise the real-time operation of the developed machine vision system.
To answer the computational constrains and to avoid the Hughes phenomenon,9,15 a large number of studies have been devoted to address the optimal decorrelation of the hyperspectral data, where the ultimate objective has been the reduction of the feature vectors that describe the materials that define the scene objects. To this end, principal component analysis (PCA),1 linear discriminant analysis (LDA),20 wavelet decomposition,2 and uniform band design (UBD),21 were previously used for hyperspectral decorrelation. While these methods proved efficient when applied to diverse practical scenarios, they have the disadvantage of having to require a substantial level of user intervention during the training stage, and in addition these decorrelation approaches need retraining if new materials are included in the classification process. To circumvent these issues, unsupervised decorrelation techniques based on fuzzy sets were proposed,9 which proved particularly efficient when applied to material classification tasks. The experimental results reported in Ref. 9 indicate that the application of decorrelation not only improves the computational efficiency, but also increases the accuracy of the classification process. Our prior investigations10 revealed that the pixel-wise features calculated after the application of data decorrelation proved insufficient to accurately model the within-class variability associated with the nonferrous fractions.9 This inadequacy of the pixel-wise features was particularly exacerbated by the specular properties and the various levels of oxidization that are characteristic for the nonferrous materials contained in the waste of electric and electronic equipment (WEEE) scrap. A robust solution resides in the combination of the spectral and spatial features with the aim of generating a descriptor that is able to factor in not only the spectral properties of the imaged materials, but also their textural characteristics as well.4,9,15,2223.–24 One approach toward spectral-spatial integration was proposed by Mercier and Lennon,25 where the authors attempted to model the texture in the hyperspectral domain by computing the marginal distribution of the wavelet coefficients using generalized Gaussian density (GDD). Other approaches implemented texture decomposition using Gabor filters,26 directional filter banks,27 Markov random fields,28 or analyzed the texture using statistical schemes based on the calculation of co-occurrence matrices.29 While these methods have shown promising results when applied to various practical applications, their major limitation is the onerous computational overhead associated with the calculation of the spectral-spatial descriptors (SSDs), a fact that rendered them as unfeasible when deployed in the development of real-time vision systems.
In a recent paper9 we detailed a novel framework where the textural and spectral features are integrated using local distributions, a concept that is well suited for sampling the spectral descriptors that are characteristic for nonferrous materials. The use of local distributions for nonferrous material sorting proved opportune from a classification performance standpoint (accuracy ), but although the computational complexity of the proposed algorithm was substantially lower than that associated with standard hyperspectral texture analysis methods,2627.28.–29 it was still too high to be feasible for real-time operation. Thus, the major goal of this paper is to redesign the process behind the calculation of the local distributions in the decorrelated hyperspectral domain that was presented in Ref. 9 with the objective of reducing the computational time to a level that offers real-time operation when the nonferrous material-sorting algorithm is included in the development of a conveyor-based WEEE recycling system. Another important objective in our research was to attain the real-time operation while maintaining the classification accuracy at a similar level when compared to the original implementation.
This paper is organized as follows. In Sec. 2, an overview of the WEEE recycling process is provided. Section 3 details the proposed material classification algorithm and its real-time implementation. Section 4 describes and discusses the experimental results, while Sec. 5 concludes the paper.
WEEE Recycling: Background Information
The most recent statistics indicate that the WEEE constitutes 4% of the total municipal waste in Europe, and it is increasing by 16 to 28% every five years.5,30 Although certain sectors of the electrical and electronic equipment (EEE) market show signs of stagnation (e.g., TV sets and large kitchen appliances), others, including information and telecommunication equipment, car electronics, and electronic toys, still experience a robust growth. In this context, it is useful to note that the European economic area (EEA) countries generate 6.5 million tons of WEEE per annum, and currently approximately 90% of this potentially hazardous waste is disposed as unsorted in generic municipal landfills. According to the current EU statistics, the WEEE is expected to increase to 12 million tons by 2015, and, as a consequence, legislation that sets specific requirements in regard to WEEE collection and recycling has been recently introduced.31 While the introduction of strict targets toward WEEE recycling had a direct impact on EEE manufacturers, as they had to adjust their environmental policies, it also opened an opportunity for companies that are active in the WEEE recycling sector.
Prior to the application of various technological recycling processes, the WEEE is shredded to allow the separation of its constituent parts. After mechanical, electrostatic, and densiometric sorting,18,32 a distinct component of the WEEE scrap is formed by nonferrous materials (stainless steel, aluminum, copper, zinc, brass, and lead) that cannot be sorted by standard mechanical recycling procedures. This fact has negative economic implications as the unsorted nonferrous materials are sold at a much lower price than the individual nonferrous fractions. Thus the availability of new technologies that would allow the robust sorting of nonferrous materials will increase considerably the profitability of the overall WEEE recycling process. To achieve this goal, the classification of the nonferrous materials has been initially carried out using the RGB data. These approaches proved inefficient when applied to the separation of several fractions of nonferrous materials (such as stainless steel, aluminum, and zinc),8 and as a result new methodologies that perform the classification in the hyperspectral domain were actively explored.9,24 While hyperspectral classification techniques proved accurate when applied to nonferrous sorting, they proved challenging when included in the development of industrial systems due to their high computational demand. To provide an insight into this issue, the specifications for WEEE recycling outlined in the SORMEN European project documentation30 indicate that a sorting speed of one ton/h is required to justify the automatic recycling process from an economical perspective. This translates into a processing speed of 50 m/min (approximately 170 camera lines/s). The algorithms based on pixel-wise classification schemes can meet this computational constraint, but they are not able to properly model the interclass variations associated with different nonferrous materials. As indicated in the introductory section of this paper, the solution to this problem resides in the integration of the spectral and spatial information in the classification process, but there are substantial computational challenges that need to be overcome. In the following sections of the paper we will detail a real-time nonferrous material classification algorithm that is able to attain the processing speed required for an industrial recycling system, while maintaining the classification accuracy at a level above 96%.
In order to devise a flexible machine vision solution, we have adopted a modular approach. In this regard, the proposed system consists of several computational modules that include the hyperspectral image acquisition and nonferrous material classification, and mechanical subcomponents that implement the WEEE vibratory delivery system, particle transport module (conveyor), and the pneumatic particle sorting mechanism. The logical arrangement of the constituent modules of the developed machine vision solution is illustrated in Fig. 1, and in the remainder of the paper we will be primarily focused on the description and analysis of the nonferrous material classification algorithm, which is the core component of the system. Details about all mechanical components of the developed machine vision solution can be found in Ref. 24.
Real-Time Nonferrous Material Classification
The computational stages associated with the proposed real-time material classification algorithm can be summarized as follows:
1. Hyperspectral data decorrelation
2. Background removal and nonferrous particle labeling
3. Data quantization and the calculation of the SSD for each nonferrous particle
4. Material classification.
For clarity, Fig. 2 presents a block diagram that describes the complete overview of the proposed real-time material classification algorithm. In this diagram, arrows illustrate the logical links between the distinct computational components of the proposed nonferrous classification algorithm. It is useful to mention that the main computational bottleneck is associated with the calculation of the SSD for each nonferrous particle (see the shaded component in Fig. 2). Thus the main objective of our work was to devise an optimized approach for the robust integration of spectral-spatial information to allow the implementation of a real-time nonferrous material-sorting system. In this sense, the new approach constructs a single descriptor for each nonferrous particle, as opposed to the approach detailed in Ref. 9, which requires the calculation of fuzzy spectral-spatial distributions for each pixel in the hyperspectral image and the application of computationally complex procedures relating to region merging and reclassification. Full details about each computational component shown in Fig. 2 will be provided in the remainder of this paper.
Hyperspectral Data Decorrelation
In the first step the unprocessed hyperspectral image is subjected to data decorrelation using an unsupervised approach based on spectral fuzzy sets.9 This approach is based on the knowledge that the spectral information varies smoothly over successive spectral bands, and as a result the characteristics associated with nonferrous materials should be sampled by groups of spectral bands rather than selective spectral bands. The data flow associated with the developed fuzzy-based decorrelation scheme is detailed in Fig. 3.
The developed hyperspectral decorrelation (see Fig. 4) involves the partitioning of the spectral domain in a predefined number of fuzzy sets, , and an energy value for each fuzzy set () is calculated as follows:1) samples the strength of the intensity signal in each fuzzy set. Using this approach, the dimensionality of each hyperspectral pixel is reduced from to (). To provide some insight into the computational efficiency attained by the proposed data decorrelation scheme, the dimensionality of the unprocessed hyperspectral data, , is 80 (i.e., consists of 80 spectral bands), and our earlier studies revealed that optimal classification results are obtained when the hyperspectral data is decorrelated using eight fuzzy sets (i.e., ) (for more details please refer to Ref. 9 where a comprehensive set of experimental results are provided).
Background Segmentation and Particle Labeling
The next step of the classification procedure involves the segmentation of the decorrelated spectral data into background and nonferrous particles. As indicated in our earlier paper, where the mechanical design of the WEEE recycling system is outlined,24 the WEEE delivery component (vibratory feeder) was carefully built to ensure a uniform placement of the nonferrous particles onto the belt of the conveyor without any particle overlaps.
Based on the hyperspectral characteristics of the nonferrous materials, it was decided to choose a belt that is coated with matte black material. The main reason behind this decision was to maximize the contrast in the hyperspectral domain between the nonferrous materials and the background information (conveyor belt) and to reduce as much as possible the occurrence of specular reflections due to the adherence of the small WEEE particles to the conveyor belt. By taking advantage of these characteristics, a sole decorrelated energy vector component (i.e., the energy component whose central wavelength corresponds to 510 nm) proved sufficient for robust background segmentation using an experimentally determined threshold. However, when the recycling system has been operated in an industrial environment, we have discovered that vertical scratches occurred on the conveyor belt due to the friction with nonferrous particles (see Fig. 5). Nonetheless, the occurrence of these vertical scratches on the conveyor belt has negative effects on the background segmentation process, as illustrated in Fig. 5(b), and to robustly eliminate these undesired features we devised a filtering approach in the Fourier domain by masking the contribution of the frequencies that are generated by the vertical scratches as shown in Fig. 5(d) and Eq. (2).Fig. 5(f)] is subjected to a computationally optimized labeling procedure. Figure 6 illustrates the image resulting from the labeling process.
Calculation of the Particle Spectral-Spatial Descriptor
After the application of the particle segmentation and labeling process, the next step involves the calculation of the spectral descriptor for each nonferrous particle. In our initial studies we have analyzed the feasibility of using pixel-wise spectral information for material classification, but the experimental results indicated a classification success rate of only 71.52% when dealing with six nonferrous fractions (aluminium, copper, brass, lead, stainless steel, and white copper). This classification accuracy clearly indicates that the spectral information provided by a single pixel is not sufficient to robustly model the significant intraclass dispersion caused by the various levels of oxidization that are characteristic for each nonferrous material. Another important factor that increased the intraclass dispersion was generated by the specular properties of the nonferrous materials and by the shadows caused by the three-dimensional (3-D) geometrical profile of each particle. While the issues inserted by the various oxidization levels can be (at least theoretically) modeled in a statistical sense, the incidence of the problems caused by the specular highlights and shadows cannot be predicted as they are caused by external factors. To alleviate the problems introduced by shadows and highlights we normalized the intensity of the decorrelated spectral information as illustrated in Eq. (3) (for more details regarding this hyperspectral intensity normalization scheme the reader can refer to Ref. 33).3.1.
However, as indicated earlier, the most important problem we confronted is the inability of the pixel-wise spectral features to properly sample the characteristics of the nonferrous materials, and a solution to this problem resides in the integration of the spatial and spectral features in a composite descriptor. Using this concept, the spectral characteristics of the nonferrous materials are sampled by the distribution of the spectral features that are calculated over a region in the image. In the context of the application detailed in this paper, we propose to calculate the distribution of the spectral-spatial information using a fuzzy-based method similar to the approach used for data decorrelation (see Sec. 3.1). This approach is motivated by the fact that several factors such as image noise, uneven illumination, and different levels of oxidization of the nonferrous materials induce undesired and unpredictable changes in the calculation of the SSDs. To alleviate these issues we mapped each component of the normalized spectral energy vectors into a new fuzzy-based representation. The process required to calculate the fuzzy distributions (histograms) entails two steps that are depicted in Figs. 7 and 8. In the first step, the intensity of each energy vector , is subjected to a quantization procedure where each spectral component is projected into a higher dimensional vector space as illustrated in Fig. 7. In this process, fuzzy sets are employed from which only two will return a value different than zero. In the second step, the SSD is calculated as shown in Fig. 8. As indicated in Fig. 8, for each pixel the quantized vectors for each energy are concatenated into a pixel-wise feature vector , while the SSD for a given region is calculated as follows,7 and 8, has several advantages that are well suited for the application targeted in this paper. First, the vector quantization procedure applied to measure the response of each energy allows the construction of soft histograms that avoid the disadvantages associated with crisp binning (i.e., as in the case when the spectral-spatial histograms would be calculated using the values). This is particularly obvious in the case of material classification, as the properties of different nonferrous fractions vary smoothly within the hyperspectral domain. Second, another important advantage resides in the fact that the compound SSD involves the simple summation of the quantized spectral response that is calculated for each pixel in the region of interest , which opens the opportunity of real-time operation. While the nonferrous particles have various sizes, prior to the classification process, each component (bin) of the SSD is normalized with respect to the number of pixels contained in the region .
Multivariate Gaussian Classifier
The histogram shown in Eq. (4) defines the distribution of the spectral energies within the area encompassed by a nonferrous particle, and this information is used as input for classification. In our implementation, the multivariate Gaussian classifier16 is proposed to perform the material classification since the dispersion of the spectral-spatial features within each class of nonferrous materials can be well approximated with normal distributions [the multivariate Gaussian classifier is optimal in the Bayes sense if the relationships between the input vectors that characterize each nonferrous material class, and the output class variables can be accurately modeled by multivariate normal distributions, as illustrated in Eq. (5)]. In this regard, a multivariate Gaussian model is created for each nonferrous material where and are the mean vector and the covariance matrix of the modeled class (in the training stage, descriptors have been used to sample the spectral characteristics for each nonferrous material).6).
The developed machine vision system consists of four distinct components: the particle feeding device, conveyor, material classification, and particle separation mechanism. The shredded WEEE mixture is automatically loaded onto a nonspecular black conveyor belt (600 mm wide) via a vibratory feeder that has been specifically designed to ensure the nonferrous materials are arranged into a thin layer prior to their arrival at the inspection line. The conveyor speed was set at 20 m/min, and the particle separation mechanism was implemented using a pneumatic part-extractor.
The WEEE mixture is defined by six nonferrous materials: white copper, aluminum, stainless steel, brass, copper, and lead (see Fig. 9). The material samples have been provided by Indumetal Recycling S.A. and IGE Hennemann Recycling GmbH, both members of the SORMEN project consortium.30 The nonferrous materials have been manually sorted by expert operators, and full information was provided about each nonferrous waste fraction. In this study the captured datasets were divided into training and testing sets, where half of the data was used for training, and the remaining half was used for testing. From each of these datasets more than 500,000 SSDs were extracted for validation purposes.
Table 1 shows the experimental results obtained by the nonferrous material sorting system detailed in this paper. The processing times reported in Table 1 were obtained when the proposed algorithm, and the algorithms described in Ref. 9, were implemented using the Microsoft Visual C++ environment (Windows XP) and executed on a standard PC that is fitted with an Intel Core 2 Duo 2.4-GHz processor and 2-GB RAM. To emphasize the computational advantages associated with the real-time classification technique presented in Sec. 3, we also provide the computational time attained by the classification method for which the SSD is calculated in predefined neighborhoods for each pixel in the image.9 As opposed to the approach presented in Ref. 9 where the classification process involves class assignments, being the number of pixels present in the image, in the implementation detailed in this paper the classification of each nonferrous particle resulting after segmentation and labeling (see Sec. 3.2) involves a single class membership assignment using Eq. (6). This is illustrated in Eq. (7) where we show the computational cost entailed by the method discussed in Ref. 9 () and the new approach detailed in Sec. 3 (). In Eq. (7) defines the computational cost associated with the extraction of the vector (see Fig. 8), is the size of the neighborhood (see Table 1), is the marginal computational cost associated with the class assignment for the unknown SSD, as indicated in Eq. (6), and denotes the number of classes.Table 1 confirm the considerable decrease in computational speed attained by the proposed approach, which allows the processing of more than two linear meters of WEEE particles per second at the conveyor speed (20 m/min). In particular, we would like to stress that the decrease in computational cost has been obtained only with a marginal drop in classification accuracy (96.87%, nonferrous classification approach detailed in this paper, and 98.36%, the neighborhood-based nonferrous material classification method described in Ref. 9, a fact that recommends the proposed solution for real-time material sorting).
|Window size||Computational speed (pixels/s)||Computational speed (m/s)||Classification accuracy (%)|
The major objective of this paper was to detail the development of an industrial compliant nonferrous material sorting system. In this paper we have reviewed the practical issues associated with the WEEE recycling process, and we have analyzed the challenges that have to be addressed during the development of an integrated nonferrous material sorting system when operated in an industrial environment. Among many practical and theoretical challenges, the computational efficiency and the accuracy of the nonferrous classification technique are critical requirements that justify the automatic recycling process from an economical standpoint. To answer these requirements, in this paper we have introduced a redesigned classification scheme in the hyperspectral domain, where the main novelty resides in the optimized computational approach that allows the real-time calculation of the SSDs in the context of nonferrous material sorting. The experimental results reveal that the proposed machine vision system is able to process the nonferrous shredded WEEE scrap at a rate of 2.28 m/s with a classification accuracy of 96.87%. The performance attained by our system exceeds the economic thresholds required for automatic WEEE nonferrous material sorting, and currently the developed machine vision system is fully evaluated in an industrial recycling environment.
We would like to thank Tecnalia, Specim and SORMEN project consortium for providing the data and the nonferrous materials that have been used in the validation of the proposed algorithm.
Artzai Picón received a MEng. degree in industrial engineering (2002) and MRes. degree in automatics electronics and control (2005) and a PhD (2009) from the University of the Basque Country, Spain. In 2000–2002, he collaborated with the Computer Image Processing Group (GTI2) in the Department of Systems Engineering and Automatics. Since then, he has been working as a researcher for Tecnalia Research Corporation where he has been involved in over 30 research projects. His research interests include image classification and segmentation and the application of these technologies to industrial tasks. In 2002 he received the Best MEng. Final Project Award from Accenture Company for the implementation of a facial recognition system; in 2006 he received the ONCE International Research and Development Award in New Technologies for the Blind and Visually Impaired.
Ovidiu Ghita received BE and ME degrees in electrical engineering from Transilvania University Brasov, Romania, and a PhD from Dublin City University, Ireland. From 1994 to 1996 he was an assistant lecturer in the Department of Electrical Engineering at Transilvania University. Since then he has been a member of the Vision Systems Group at Dublin City University (DCU), and currently he holds a position of DCU-Research Fellow. She has authored and co-authored over 90 peer-reviewed research papers in areas of instrumentation, range acquisition, machine vision, texture analysis, and medical imaging.
Aranzazu Bereciartua received her physics degree from the University of the Basque Country, Spain (1999), and her electronics engineer degree from the University of the Basque Country, Spain (2000). Since 2000 she has been employed by Fundación Robotiker (now Tecnalia Research & Innovation), when she joined the Image Processing Group as a software developer. Since 2005 she holds the position of project manager in this company. She has 11 years’ experience in machine vision, image processing algorithms, sensors and advanced automation, artificial intelligence and intelligent interfaces. She has managed and developed applications from the conception to the final installation stage at users’ sites, with particular contributions in system design, SW development, and HW & SW integration. She has taken part in many local, national, and European projects during which time she contributed to one patent and authored several publications in the field of machine vision. Currently, she is working toward the completion of her doctoral thesis in biomedical image processing.
Jone Echazarra received her electrical engineer degree from the Engineering School of Bilbao (1989). In 1990 she joined Philips CFT (Eindhoven, Netherlands) where she worked within the EU project ESPRIT 2017: Automated process and assembly inspection by 3D-Vision. Since 1991 she has been employed by Fundación Robotiker (now Tecnalia Research & Innovation) where in addition to the researcher position within the Computer Vision Group she also serves as project and market manager. She has contributed to several conferences in the field of computer and machine vision, and she served as a coordinator for several EU projects.
Paul F. Whelan received his BEng. (Hons) degree from NIHED, MEng. from the University of Limerick, and his PhD (computer vision) from the University of Wales, Cardiff, UK (Cardiff University). In 1985 to 1990 he was employed by Industrial and Scientific Imaging Ltd. and later Westinghouse (WESL), where he was involved in the research and development of high-speed computer vision systems. He was appointed to the School of Electronic Engineering, Dublin City University (DCU), in 1990 and is currently professor of computer vision (personal chair). He founded the Vision Systems Group in 1990 and the Centre for Image Processing & Analysis in 2006 and currently serves as its director. As well as publishing over 150 peer-reviewed papers, he has co-authored two monographs and co-edited three books. His research interests include image segmentation and its associated quantitative analysis with applications in computer/machine vision and medical imaging. He is a senior member of the IEEE, fellow of IET, chartered engineer, and a member of IAPR. He served as a member of the governing board (1998 to 2007) of the International Association for Pattern Recognition (IAPR), a member of the International Federation of Classification Societies (IFCS) council and president (1998 to 2007) of the Irish Pattern Recognition and Classification Society (IPRCS). Prof. Whelan is a HEA-PRTLI (RINCE, NBIP) funded principal investigator.
Pedro M. Iriondo received a degree in industrial engineering in 1981 and began his professional activity working in different companies. In 1986 he founded the company Adicorp dedicated to the research, integration, and development of industrial vision systems. In 2001 he received a PhD in industrial engineering—electronics and control engineering—from the University of the Basque Country UPV-EHU, Spain. Since 1989 he has been an associate professor with the Department of Automatic Control and Systems Engineering, ETSI of Bilbao, University of the Basque Country UPV-EHU. Currently, he is the assistant director of the School of Engineering of Bilbao, and he is responsible for relations with private companies. His research interest is focused in the fields of image processing, face detection and recognition, image transmission and stochastic systems.