This paper presents an efficient system for recovering the structural properties of objects. We build adaptive maps for interpreting the object structure. The System is decomposed into four major modules, the sensing interface acquires visual data and performs low and intermediate vision tasks for enhancing the acquired image sequences. A knowledge base contains different visual primitives and the possible exploratory actions. The map building and decision making module gets its input from the sensing interface and utilizes the different predicates that are stored in the knowledge base in order to resolve possible uncertainties. The decisions made in the map building and decision making module are then utilized by the controller module which resolves possible inconsistencies due to physical limitations of the sensing devices. The above process is repeated until the map contains a minimal number of interpretations that cannot be reduced further.