17 May 2016 Spectral signature verification using statistical analysis and text mining
Author Affiliations +
In the spectral science community, numerous spectral signatures are stored in databases representative of many sample materials collected from a variety of spectrometers and spectroscopists. Due to the variety and variability of the spectra that comprise many spectral databases, it is necessary to establish a metric for validating the quality of spectral signatures. This has been an area of great discussion and debate in the spectral science community. This paper discusses a method that independently validates two different aspects of a spectral signature to arrive at a final qualitative assessment; the textual meta-data and numerical spectral data. Results associated with the spectral data stored in the Signature Database1 (SigDB) are proposed. The numerical data comprising a sample material’s spectrum is validated based on statistical properties derived from an ideal population set. The quality of the test spectrum is ranked based on a spectral angle mapper (SAM) comparison to the mean spectrum derived from the population set. Additionally, the contextual data of a test spectrum is qualitatively analyzed using lexical analysis text mining. This technique analyzes to understand the syntax of the meta-data to provide local learning patterns and trends within the spectral data, indicative of the test spectrum’s quality. Text mining applications have successfully been implemented for security2 (text encryption/decryption), biomedical3 , and marketing4 applications. The text mining lexical analysis algorithm is trained on the meta-data patterns of a subset of high and low quality spectra, in order to have a model to apply to the entire SigDB data set. The statistical and textual methods combine to assess the quality of a test spectrum existing in a database without the need of an expert user. This method has been compared to other validation methods accepted by the spectral science community, and has provided promising results when a baseline spectral signature is present for comparison. The spectral validation method proposed is described from a practical application and analytical perspective.
© (2016) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Mallory E. DeCoster, Mallory E. DeCoster, Alexe H. Firpi, Alexe H. Firpi, Samantha K. Jacobs, Samantha K. Jacobs, Shelli R. Cone, Shelli R. Cone, Nigel H. Tzeng, Nigel H. Tzeng, Benjamin M. Rodriguez, Benjamin M. Rodriguez, "Spectral signature verification using statistical analysis and text mining", Proc. SPIE 9840, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXII, 984025 (17 May 2016); doi: 10.1117/12.2225214; https://doi.org/10.1117/12.2225214


Science operations at Gemini Observatory
Proceedings of SPIE (July 14 2016)
Discrimination requirements model
Proceedings of SPIE (July 04 1995)
Estimate LAMOST hot star's parameters by POLLUX
Proceedings of SPIE (September 23 2012)
Coherent integration of NPOI phase closure data on Altair
Proceedings of SPIE (October 19 2004)
XMM-Newton (cross)-calibration
Proceedings of SPIE (October 10 2004)

Back to Top