Simulation is used in robot navigation for testing control algorithms such as obstacle avoidance and path planning. Simulation is also being used for generating expectations to guide sensory processing for robots operating in the real world. In this paper, we present the tradeoffs in designing an environment model for outdoor environments. The models for outdoor environments are significantly different from indoor environment models. Outdoor environments are inherently unstructured due to changing lighting conditions, variation in the form of objects, and the dynamic nature of the environment. We present the design tradeoffs involved in building models of such environments, from the perspective of simulating passive sensors like vision. We point out that a powerful approach to outdoor navigation is to have a detailed model of the environment that can provide expectations both in terms of the spatial location of scene entities and the operators suitable for detecting these entities in an image.