Damage diagnosis is a problem that can be addressed at many levels. Stated in its most basic form, the objective is to ascertain simply if damage is present or not. In a statistical pattern recognition paradigm of this problem, the philosophy is to collect baseline signatures from a system to be monitored and to compare subsequent data to see if the new 'pattern' deviates significantly from the baseline data. Unfortunately, matters are seldom as simple as this. In reality, structures will be subjected to changing environmental and operational conditions that will affect measured signals. In this case, there may be a wide range of normal conditions, and it is clearly undesirable to signal damage simply because of a change in the environment. In this paper, a unique combination of time series analysis, neural networks, and statistical inference techniques is developed for damage classification explicitly taking into account these natural variations of the system in order to minimize false positive indication of true system changes.