A scene interpretation system can be partitioned into three levels of processing: low level, mid level, and high level. We describe here a systematic approach to scene interpretation that embeds these three levels of processing in a structured hierarchical blackboard architecture. Each blackboard level contains modular knowledge sources which perform numeric and/or symbolic processing. The low level processing blackboard is embedded as one of the knowledge sources within the mid level processing blackboard which is in turn embedded as a knowledge source within the high level processing blackboard. The benefits of this scene interpretation system architecture are efficient bi-directional flow of control between processing levels, modularized multispectral/multisensor image processing, and information feedback for performance improvement and for spatial/temporal tracking.