Tracking Issue: Scheme Design with Editions

[connortsui20]

This is a tracking issue to determine the design of Schemes with respect to Vortex editions.

High-level definitions:

• A scheme is an in-memory compression strategy for creating array encodings trees.
• An edition is a set of allowed wire IDs of encodings that are allowed to be serialized to disk.

Motivation

With the recent edition work, we have realized that schemes are linked to editions. Even though they are theoretically separate abstractions, practically speaking they are tied together because we do not want our compression strategies (schemes) to produce a bunch of encodings that are not allowed by the editions.

If editions are the "allow list" of wire IDs, we can enforce this in 3 places:

1. When serializing, check the allow list, and if it is not allowed, hard error.
2. Inside the cascading compressor, if we produce a disallowed encoding, hard error.
3. While constructing the cascading compressor, choose ONLY schemes that will not produce a disallowed encoding (this includes scheme versions.

Hopefully it is clear why the third is preferable in a lot of cases: we do not want to do all the work of sampling and compressing an array just to have to hard error at the end. So we would like to know what schemes the compressor is allowed to use before we start compressing things.

This means that we need to figure out 2 independent (or what I think are independent) things:

1. What does the writer want to consider using (like what schemes are we enabling / prefer)?
2. What output is the writer permitted to produce (what do the editions allow/disallow)?

In simpler terms, 1 is what a writer "wants", and 2 is what a writer "can".

Notice how 1 is additive, and 2 is subtractive. I think this is a nice distinction for schemes vs editions.

Once the writer has figured these things, it can move on to step 3, where it should know ALL of the allowed schemes, and it can just pass them all to the cascading compressor.

Design

I think there are 2 possible designs here.

1. Use a builder (like our current BtrBlocksCompressorBuilder) for 1, pass in the allow list of wire IDs as a required "filter" for 2, then call build for 3.
2. Use a session registry for 1, and then for 2 and 3 use BtrBlocksCompressorBuilder::from_session(SESSION), since the session also has that allow list. (also I think we should change the name)

Note that no matter what, we should not have a big static default list of schemes (like we have right now) because that does not compose well at all.

Design

I don't think it makes that much sense to have both a session registry and a builder here because it seems to be sufficient to choose one or the other.

We talked offline about how the builder permeating everywhere with hardcoded filter and additive methods (only_cuda_compatible, with_compact) is very fragile, and I don't think you can avoid that with a builder.

So it seems like we want to go with option 2. That also has the benefit of registering schemes right next to where the encoding it will produce is registered.

There is also a bit of handwaving here, like does registering a scheme mean that it should be enabled for compression? Or do we need another step to filter things out? Personally, I do not like the idea that we would register everything that we possibly can, and then separately filter out schemes before we start compressing, imo it makes more sense that scheme registry is all or nothing.

Put another way: our "wants" (the schemes that we want to compress with) should be a like a list of things we want, not a list of things we don't want from the list of all possible things we can have (which is closer to what we have now, where we subtract from the entire universe of schemes).

Also note that we probably need to figure out how to do scheme versioning, though this is an orthogonal problem.

Steps

• Experiment: compose scheme registration with output editions #10040 (this was really bad)
• Register compression schemes on the session #10034 (this is better)

Unresolved questions

• How should configured schemes relate to the stateful scheme work in Tracking Issue: Stateful Schemes #8434?
• Does vortex-btrblocks retain a distinct responsibility after its schemes move into their owning packages?

[mhk197]

There are some significant cases in the current codebase where we exclude or enable schemes after the session is initialized. An illustrative example is here. I dont think the solution is to create another session, we need to retain the builder's ability to exclude schemes in cases like these.

[connortsui20]

@mhk197 I think that is the only hardcoded place that we exclude a scheme. Everywhere else, it should be possible to configure the session correctly. So imo that is fine to be a private impl detail (considering Dict is also one of our built-in schemes), but it should not be available to users. (see some of my other comments on #10034)

[mhk197]

There is also the mode question:

• Python write uses compact based on a flag
• Conversion for tui uses compact based on a flag
• Used all over benchmarks (easier to change since editions do not matter as much there)

For cuda, I think we can make it users responsibility to only register cuda-compatible schemes.

[connortsui20]

ok so it seems like we are going to do this:

• register all of the schemes on the session
• pass it to a BtrBlocksCompressorBuilder
• the builder can add and remove schemes (like a CompressionStrategy)
• the builder can have presets for things like compact and cuda

This seems like the simplest strategy we can do for now, and if there are issue we can revisit this.