feat(optimize): select the best structured transform

[cgasgarth]

What changed

Structured JSON compression now performs a verified best-of-N search instead of shipping the first codec that clears the saving gate.

• Evaluates Columnar, Normalize, Nested, and Doc independently through encode, admission, detector, and cost gates.
• Chooses the admitted candidate with the fewest configured-model tokens; ties break by transform ID, then wire bytes, for deterministic cache-friendly output.
• Continues after an admission or detector rejection, surfacing a refusal only after the structured search is exhausted.
• Rebuilds every registered built-in structured transform when Optimizer::with_counter() installs a tokenizer. (Shred already carried one.)
• Makes Doc choose its embedded normalized/nested table and its table-admission gate by token count rather than String::len().

No wire format or decoder behavior changes.

Why

Columnar was always tried first, so it could mask a cheaper Nested or Normalize representation for overlapping record shapes. Doc also made byte-based decisions inside a token-priced optimizer. This lets SecondWind consistently choose the cheapest proven readable wire for the active model, without changing fidelity proofs or detector coverage.

cl100k inline benchmark

Measured on the checked-in compression corpus with offload disabled, comparing upstream main with this branch. Counts are the emitted inline wire's cl100k tokens.

Corpus case	Input tokens	main	This PR	Improvement
High-cardinality record array	19,502	9,551 (Columnar)	9,548 (Nested)	3 fewer (0.03%)
Low-cardinality record array	15,673	4,741 (Columnar)	4,538 (Nested)	203 fewer (4.3%)
Flat object	23,997	23,997 (verbatim)	23,997 (verbatim)	—
Small record array	25	24 (Columnar)	24 (Columnar)	—

The low-cardinality case improves total input reduction from 69.8% to 71.0% while remaining byte-exact on recovery.

Validation

• cargo test --workspace
• cargo test -p secondwind-optimize --features tiktoken
• cargo clippy --workspace --all-targets
• cargo fmt --check
• New regression tests cover later-candidate recovery after admission/detector rejection, token-based ranking, deterministic ties, overlapping built-ins, and Doc's token-aware embedded selection.

[hardy30894]

Good catch.
LGTM!

[cgasgarth]

Happy to contribute back, using the package in github.com/cgasgarth/clodex

[hardy30894]

@cgasgarth Awesome! Seeing savings first time in real action!