Algorithm development for automated outlier detection and background noise reduction during NIR spectroscopic data processing

David Abookasis; Jerome J. Workman

doi:10.1117/12.892698

16 September 2011 Algorithm development for automated outlier detection and background noise reduction during NIR spectroscopic data processing

David Abookasis, Jerome J. Workman

Author Affiliations +

Proceedings Volume 8137, Signal and Data Processing of Small Targets 2011; 813703 (2011) https://doi.org/10.1117/12.892698
Event: SPIE Optical Engineering + Applications, 2011, San Diego, California, United States

Abstract

This study describes a hybrid processing algorithm for use during calibration/validation of near-infrared spectroscopic signals based on a spectra cross-correlation and filtering process, combined with a partial-least square regression (PLS) analysis. In the first step of the algorithm, exceptional signals (outliers) are detected and remove based on spectra correlation criteria we have developed. Then, signal filtering based on direct orthogonal signal correction (DOSC) was applied, before being used in the PLS model, to filter out background variance. After outlier screening and DOSC treatment, a PLS calibration model matrix is formed. Once this matrix has been built, it is used to predict the concentration of the unknown samples. Common statistics such as standard error of cross-validation, mean relative error, coefficient of determination, etc. were computed to assess the fitting ability of the algorithm Algorithm performance was tested on several hundred blood samples prepared at different hematocrit and glucose levels using blood materials from thirteen healthy human volunteers. During measurements, these samples were subjected to variations in temperature, flow rate, and sample pathlength. Experimental results highlight the potential, applicability, and effectiveness of the proposed algorithm in terms of low error of prediction, high sensitivity and specificity, and low false negative (Type II error) samples.

Citation Download Citation

David Abookasis and Jerome J. Workman "Algorithm development for automated outlier detection and background noise reduction during NIR spectroscopic data processing", Proc. SPIE 8137, Signal and Data Processing of Small Targets 2011, 813703 (16 September 2011); https://doi.org/10.1117/12.892698

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available