In this paper the architecture of an autonomous human behavior detection system is presented. The proposed system architecture is intended for Security and Safety surveillance systems that aim to identify adverse events or behaviors which endanger the safety of people or their well-being. Applications include monitoring systems for crowded places (Malls, Mass transport systems, other), critical infrastructures, or border crossing points. The proposed architecture consists of three modules: (a) the event detection module combined with a data fusion component responsible for the fusion of the sensor inputs along with relevant high level metadata, which are pre-defined features that are correlated with a suspicious event, (b) an adaptive learning module which takes inputs from official personnel or healthcare personnel about the correctness of the detected events, and uses it in order to properly parameterise the event detection algorithm, and (c) a statistical and stochastic analysis component which is responsible for specifying the appropriate features to be used by the event detection module. Statistical analysis estimates the correlations between the features employed in the study, while stochastic analysis is used for the estimation of dependencies between the features and the achieved system performance.