Close the vortex-java vs vortex-jni file-size gap on real-world data (nyc-311: 1.62x)

[dfa1]

Summary

Re-imported the Raincloud nyc-311 slug (18.49M rows, 42 mostly-Utf8 columns) through the current writer after the #287 FSST rewrite: vortex-java=2854.88MB vs the corpus reference (vortex-jni/Python-bindings-written) 1762.17MB — 1.62x larger. Diffing the two files' layout trees (vortex-cli inspect) surfaced two concrete, well-substantiated gaps, not a regression — FSST itself is working correctly (see #298, a separate inspector display issue that initially made this look like FSST wasn't being used at all).

Finding 1 — global-dict cardinality cap (GLOBAL_DICT_MAX_CARDINALITY = 2048) is too conservative

From ADR 0021, the cap was set deliberately to bound buffering memory during the OOM fix — not because 2048 is an inherent ceiling for when a global dictionary stops being worthwhile.

The reference file uses a genuinely global dictionary (one dict spanning all 18.49M rows) for columns whose distinct-value count is well past our cap:

Column	Distinct values (reference)	Currently admitted to global dict?
Additional Details	9,388	No — falls back to per-chunk
Street Name	20,082	No
Cross Street 1	27,634	No
Cross Street 2	23,674	No
Intersection Street 1	16,764	No
Intersection Street 2	17,691	No
Landmark	11,795	No

Java's admission gate rejects any column exceeding 2048 distinct values from global dict entirely, pushing these into costlier per-chunk fallbacks (FSST, or worse).

The underlying storage (short[] per-chunk code buffering) already supports cardinality up to ~32,767 without any structural change (see ADR 0021's "Risks to manage" section, which already flags this exact tension: "if the constant is ever raised past either bound, both the buffer element type and codePTypeForSize selection need revisiting together" — for a raise up to ~32767 the short[] buffer type itself does not need to change). Raising GLOBAL_DICT_MAX_CARDINALITY (with the accompanying codePTypeForSize/wire-code-width check ADR 0021 calls out) is a small, low-risk change with likely broad real-world impact, given how many text columns in this file land in the 1K–30K distinct-value range.

Finding 2 — DictEncodingEncoder's value pool is never compressed

writer/src/main/java/io/github/dfa1/vortex/writer/encode/DictEncodingEncoder.java#encodeUtf8 hardcodes the dictionary's distinct-values child as raw vortex.varbin:

EncodeNode valuesNode = new EncodeNode(EncodingId.VORTEX_VARBIN,
        MemorySegment.ofArray(varBinMetaBytes),
        new EncodeNode[]{offsetsNode},
        new int[]{0});

It is never cascaded through FSST (or any other candidate). The reference file's layout tags show vortex.dict and vortex.fsst together on the same columns — i.e. Rust dict-deduplicates rows and FSST-compresses the resulting distinct-value pool, capturing both row-level duplication and substring redundancy (shared "STREET"/"AVENUE"/borough names across otherwise-distinct dictionary entries). This is additive on top of Finding 1 — even a column that stays under a raised cardinality cap still pays full uncompressed-varbin cost for its values pool today.

Secondary, less-isolated factor

Rust's physical chunk granularity for the affected columns (32,768–65,536 rows) is noticeably finer than Java's (131,072 rows per flat layout segment for this same file/import path). Finer chunking likely also shifts some per-chunk cost-competition outcomes (dict cardinality gates, FSST training quality) independent of the global-dict question above, but is harder to isolate and is not the primary lever — noted here for completeness, not as a first action item.

Suggested next steps (not started — analysis only, filed per user request)

1. Raise GLOBAL_DICT_MAX_CARDINALITY in VortexWriter (with the wire-code-width follow-through ADR 0021 flags) and re-measure nyc-311's file size — expected to be the highest-leverage single change.
2. Consider cascading DictEncodingEncoder.encodeUtf8's values child through the same cost-based competition (FSST/varbin) that non-dict Utf8 columns already get, instead of hardcoding vortex.varbin.
3. Optionally revisit default physical chunk size for text-heavy columns, once 1–2 are measured.

Where

• writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java — GLOBAL_DICT_MAX_CARDINALITY constant, admission gate
• writer/src/main/java/io/github/dfa1/vortex/writer/encode/DictEncodingEncoder.java — encodeUtf8's hardcoded vortex.varbin values node
• adr/0021-cardinality-bounded-global-dict-buffering.md — prior design context and the risk this issue's Finding 1 was already flagged against