Informix under control

[srina]

INFORMIX UNDER CONTROL
divide any data flow into four stages: the produce stage, the reorder stage, the process
stage, and the consume stage. In the context of query processing, Produce represents
an access method generating records. Reorder reorders the items according to the dynamically
changing preferences of the consumer. Process is the set of operations that
are applied “downstream” to the records—this could involve query plan operators like
joins, shipping data across a slow network, rendering data onto the screen in data visualization,
etc. Consume captures the user think-time, if any—this is mainly for interactive
interfaces such as spreadsheets or data visualization. Since all these operations can go on
concurrently, we exploit the difference in throughput between the produce stage and the
process or consume stages to permute the items. For disk-based data sources, Produce
can run as fast as the sequential read bandwidth, whereas process may involve several random
I/Os which are much slower (Gray and Graefe, 1997). While the items sent out so far
are being processed/consumed, reorder can take more items from produce and permute
them.
The reorder operator tries to put as many interesting items as possible onto a mainmemory
buffer, and the process operator issues requests to get items from the buffer.
Process decides which item to get based on its performance goals. Reorder uses the
time gap between successive gets from the buffer (which may arise due to processing or
consumption time) to populate the buffer with more interesting items. It does this either
by using an index to fetch interesting items, or by aggressively prefetching from the input,
spooling uninteresting items onto an auxiliary disk. Policies for management of the buffer
and organization of the auxiliary disk are described in more detail in Raman et al. (1999b);
the basic idea is to evict least-interesting items from the buffer, and place them into chunks
on disk of records from the same group.
If the reorder operator can get records much faster than they can be processed, then the
reordering has two phases. In the first phase, reorder continually gets data, and tries to keep
the buffer full of interesting items in the appropriate ratios, carefully spooling uninteresting
items to chunks on the side disk. The second phase occurs when there is no more data to
get; at this point, reorder simply enriches the buffer by fetching chunks of interesting tuples
of interest from the side disk.