Runtime Support for Disk Based Sparse Multidimensional Datastructures


Joel Saltz

As computational power and storage capacity increase, the processing and
analysis of large disk based datasets play an increasingly important
part in many domains of scientific research. Typical examples of very
large scientific datasets include long running simulations of
time-dependent phenomena that periodically generate snapshots of their
state, archives of raw and processed remote
sensing data and archives of radiological and microscopic images.
These datasets are usually multi-dimensional and frequently are stored
using sparse or multiresolution datastructures. The data dimensions can
be spatial coordinates, time, or varying experimental conditions such
as temperature, velocity or magnetic field.

We have developed a parallel runtime support library (T2) to support
access to and processing of large sparse multidimensional datasets. T2
provides support for operations that including data clustering and
declustering, associative datastructure access procedures such as range
queries, a class of procedures to specify broadly useful classes of data
aggregation operations, index generation, memory management and
computational scheduling. T2 achieves its primary advantage from the
ability to integrate data retrieval and processing for a wide variety of

applications, and from the ability to maintain and jointly process
multiple datasets with different underlying regular or irregular grid
based datastructures. We are currently using T2 as runtime support for
databases that support access and processing of earth science,
microscopy and scientific computing datasets. We also are beginning the
process of adapting T2 to support efficient datastructure access in
object relational database systems.