In this paper we propose a technique called storage-aware spatial prefetching that can provide significant performance
improvements for out-of-core visualization. This approach is motivated by file chunking in which a
multidimensional data file is reorganized into multidimensional sub-blocks that are stored linearly in the file.
This increases the likelihood that data close in the n-dimensional volume represented by the file will be closer
together in the physical file. Chunking has been demonstrated to improve the typical access to such data, but it
requires a complete re-organization of the file and sometimes efficient access is only achieved if multiple different
chunking organizations are maintained simultaneously. Our approach can be thought of as on-the-fly chunking,
but it does not require physical re-organization of the data or multiple copies with different formats. We also
describe an implementation of our technique and provide some performance results that are very promising.
Visualization of multidimensional data presents special challenges for the design of efficient out-of-core data access. Elements that are nearby in the visualization may not be nearby in the underlying data file, which can severely tax the operating system’s disk cache. The Granite Scientific Database System can address these problems because it is aware of the organization of the data on disk, and it knows the visualization method’s pattern of access. The access pattern is expressed using a toolkit of iterators that both describe the access pattern and perform the iteration itself. Because our system has knowledge of both the data organization and the access pattern, we are able to provide significant performance improvements while hiding the details of out-of-core access from the visualization programmer. This paper presents a brief description of our disk access system placing special emphasis on the benefits offered to a visualization application. We describe a simple demonstration application that shows dramatic performance improvements when used with the 39GB Visible Woman Dataset.