Skip to content

feat: leaf expression markers — recovery reads authored leaves, never guesses - #305

Merged
hussainsultan merged 2 commits into
fix/join-leaf-recovery-collisionsfrom
feat/leaf-expression-markers
Aug 23, 2026
Merged

feat: leaf expression markers — recovery reads authored leaves, never guesses#305
hussainsultan merged 2 commits into
fix/join-leaf-recovery-collisionsfrom
feat/leaf-expression-markers

Conversation

@hussainsultan

@hussainsultan hussainsultan commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

The idea

Stacked on #304. to_tagged serializes one lowered expression plus metadata, and recovery re-derives each model's table from the lowered plan (join splitting + base-relation walking) — guessing where lowering ends and the authored expression begins. Every failure in this family was that guess going wrong: __bsl_jk_ seam projections, shaped views discarded, aggregate-grain views unrecoverable, query entries losing their shaped base.

This PR removes the guessing: to_tagged stamps each leaf model's authored table expression with a marker tag inside the lowered payload (__bsl_leaf__, {"leaf": <model name>}), and recovery returns the marked subtree verbatim before any heuristic runs.

  • _collect_leaf_tables maps each leaf SemanticTableOp to its table op (descending join wrappers' _source_join).
  • _mark_leaf_tables wraps every structural occurrence via a node-equality rewrite — markers land wherever lowering placed the leaf: under rename projections, under pre-aggregation legs, under a query's Aggregate.
  • Markers are hashing tags: payload-light, profiles ride along, and they serialize through xorq's YAML build path unchanged.

What becomes first-class (catalog-round-trippable)

  1. Shaped deferred views — the star-schema doctrine: conformed dims/facts shaped in bare xorq, thin naming layer on top.
  2. to_semantic_table(table.group_by(...).aggregate(...)) — aggregate-grain fact models (dimensional-modeling aggregate fact tables at declared grains). Verified in-memory and through the full xorq build → load_expr → .ls.builder disk path with correct values.
  3. Query entries over shaped bases — the strict-xfail gap from fix: recover join leaves poisoned by internal __bsl_jk_ temporaries #304's second commit now passes and is promoted to a regular test.

Compatibility

  • Payloads without markers (pre-change, or a leaf whose table op was rewritten by lowering so no node matches) fall back to the existing heuristics unchanged — covered by an explicit strip-the-markers test.
  • Old readers ignore the inner tags (their walks pass through Tag nodes).

Upstream note (xorq, untouched here)

Plain-callable replacers in ibis's replace() must recreate nodes from _kwargs themselves for child substitutions to propagate (Pattern/Mapping replacers get this via _coerce_replacer; bare callables do not). The marker rewrite does this correctly — but to_tagged's pre-existing replace_read_parquet callable does not, and likely no-ops on nested rewrites. Worth an xorq-side look.

Testing

  • 11 leaf-recovery tests pass (shaped single/join/disk, grouped-grain in-memory + disk, query-entry-over-shaped-base, marker-stripped fallback, guard-message, look-alike column).
  • Serialization battery (convert/rebuild/string-serialization/tag-handler/recovery): 115 passed.
  • Full suite in a fresh isolated env: 1307 passed, 6 failed — all 6 reproduce on the base commit (optional-dependency environment failures in langgraph/mcp/flavor-routing), zero regressions from this change.

🤖 Generated with Claude Code

Conflicts resolved (merge commit 67396ee, no history rewrite)

Base gained 4a4e26f (sibling-wipe serialization fix + new tests for tagging already-aggregated join queries). The conflict was append-vs-append in the recovery test file; resolved keeping BOTH test sets — 13 tests, all passing.

One honest finding from the merge: the base's new test_join_many_aggregate_query_fails_loud_not_silently_wrong still passes with markers. Pre-agg compilation of a tagged join_many QUERY rebuilds the leaf nodes into partial-aggregate/key-bridge legs, so no structural occurrence of the authored leaf survives for the node-equality marker rewrite to find — that case remains a loud, actionable failure (tag the bare model instead; supported and tested). Markers DO rescue single-table query entries over shaped bases (test_query_entry_recovers_the_shaped_base, previously strict-xfail). Making fan-out query entries recoverable would need markers applied inside the pre-agg compiler itself — possible follow-up.

hussainsultan and others added 2 commits August 23, 2026 17:36
… guesses

The last of the leaf-recovery family. to_tagged serialized ONE lowered
expression plus metadata, and recovery re-derived each model's table from
the lowered plan (join splitting + base-relation walking) — guessing where
lowering ends and authored expression begins. Every failure in this family
was that guess going wrong: seam projections with __bsl_jk_ temporaries,
shaped views discarded, aggregate-grain views unrecoverable, query entries
losing their shaped base.

to_tagged now stamps each leaf model's AUTHORED table expression with a
marker tag inside the lowered payload (__bsl_leaf__, metadata
{"leaf": <model name>}): _collect_leaf_tables maps leaf SemanticTableOps to
their table ops (descending join wrappers' _source_join), and
_mark_leaf_tables wraps every structural occurrence via a node-equality
rewrite — so markers land wherever lowering placed the leaf: under rename
projections, under pre-aggregation legs, under a query's Aggregate. Markers
are hashing tags: payload-light, profiles ride along, they serialize
through xorq's YAML build path unchanged.

Recovery (_find_marked_leaf) returns the marked subtree verbatim before any
heuristic runs. This makes first-class, catalog-round-trippable:

- shaped deferred views (star-schema doctrine),
- to_semantic_table(table.group_by(...).aggregate(...)) — aggregate-grain
  fact models (dimensional-modeling aggregate fact tables),
- query entries over shaped bases (the previously strict-xfail gap — that
  test now passes and is promoted to a regular test).

Compatibility: payloads without markers (pre-change, or a leaf whose table
op was rewritten by lowering so no node matches) fall back to the existing
heuristics unchanged — covered by an explicit strip-the-markers test. Old
readers ignore the inner tags (their walks pass through Tag nodes).

Note on the rewrite: plain-callable replacers in ibis's replace() must
recreate nodes from _kwargs themselves for child substitutions to
propagate (Pattern/Mapping replacers get this for free) — the replacer
here does; to_tagged's pre-existing replace_read_parquet callable does
not and likely no-ops on nested rewrites (upstream xorq note, untouched).

Tests: 11 recovery tests pass incl. grouped-grain in-memory + full disk
round-trip; serialization battery 115 passed; full suite in a fresh env:
1307 passed, 6 failed — all 6 reproduce on the base commit (optional-dep
environment failures: langgraph/mcp/flavor-routing), zero regressions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts:
#	src/boring_semantic_layer/tests/test_xorq_join_leaf_recovery.py
@hussainsultan
hussainsultan marked this pull request as ready for review August 23, 2026 21:51
@hussainsultan
hussainsultan merged commit 6a68bcf into fix/join-leaf-recovery-collisions Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant