Improve index memory management

[albe]

Currently an index preallocates an array of the length of the index and then fills it with data on demand. Since the internal array is untyped, memory is allocated just for the structure. Still, for large indices this means a whole lot of memory being kept allocated unused in a couple of use cases, most notably the write-only scenario.

Optimally an index would only keep a (configurable) fixed upper bound of data in memory, then fill that on demand.
This could be a good scenario for a ring buffer.

Also, optimally the internal array buffer would be a typed array of (u)int32 to have index data in a contiguous memory block. This could potentially optimize index entry/buffer translation since the entry would just be a typed view on the underlying buffer and no copying on read would be involved.

[albe]

Index range reading should return a generator and work with the same semantics as partition reading. Hence, the underlying file abstraction could be shared between index and partition.

[albe]

See https://gist.github.com/albe/39c7b79f46daa49d2cf373ffab3c4513 -ugh

[albe]

The feedback on nodejs/help repo suggests this is to be expected, as typed arrays do a bit more.
After some testing with creating a custom implementation of a "buffer view entry" the single access use case is faster than the current implementation by a factor of 2, but slower by a factor of two when a second access happens (and hence likely even more for further accesses - i.e. it behaves bad for "cache hits" in the index reader). So the buffer reads need to be memoized eagerly (lazy adds a condition in the access path).