feat: leaf expression markers — recovery reads authored leaves, never guesses - #305
Merged
hussainsultan merged 2 commits intoAug 23, 2026
Conversation
… guesses
The last of the leaf-recovery family. to_tagged serialized ONE lowered
expression plus metadata, and recovery re-derived each model's table from
the lowered plan (join splitting + base-relation walking) — guessing where
lowering ends and authored expression begins. Every failure in this family
was that guess going wrong: seam projections with __bsl_jk_ temporaries,
shaped views discarded, aggregate-grain views unrecoverable, query entries
losing their shaped base.
to_tagged now stamps each leaf model's AUTHORED table expression with a
marker tag inside the lowered payload (__bsl_leaf__, metadata
{"leaf": <model name>}): _collect_leaf_tables maps leaf SemanticTableOps to
their table ops (descending join wrappers' _source_join), and
_mark_leaf_tables wraps every structural occurrence via a node-equality
rewrite — so markers land wherever lowering placed the leaf: under rename
projections, under pre-aggregation legs, under a query's Aggregate. Markers
are hashing tags: payload-light, profiles ride along, they serialize
through xorq's YAML build path unchanged.
Recovery (_find_marked_leaf) returns the marked subtree verbatim before any
heuristic runs. This makes first-class, catalog-round-trippable:
- shaped deferred views (star-schema doctrine),
- to_semantic_table(table.group_by(...).aggregate(...)) — aggregate-grain
fact models (dimensional-modeling aggregate fact tables),
- query entries over shaped bases (the previously strict-xfail gap — that
test now passes and is promoted to a regular test).
Compatibility: payloads without markers (pre-change, or a leaf whose table
op was rewritten by lowering so no node matches) fall back to the existing
heuristics unchanged — covered by an explicit strip-the-markers test. Old
readers ignore the inner tags (their walks pass through Tag nodes).
Note on the rewrite: plain-callable replacers in ibis's replace() must
recreate nodes from _kwargs themselves for child substitutions to
propagate (Pattern/Mapping replacers get this for free) — the replacer
here does; to_tagged's pre-existing replace_read_parquet callable does
not and likely no-ops on nested rewrites (upstream xorq note, untouched).
Tests: 11 recovery tests pass incl. grouped-grain in-memory + full disk
round-trip; serialization battery 115 passed; full suite in a fresh env:
1307 passed, 6 failed — all 6 reproduce on the base commit (optional-dep
environment failures: langgraph/mcp/flavor-routing), zero regressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # src/boring_semantic_layer/tests/test_xorq_join_leaf_recovery.py
hussainsultan
marked this pull request as ready for review
August 23, 2026 21:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The idea
Stacked on #304.
to_taggedserializes one lowered expression plus metadata, and recovery re-derives each model's table from the lowered plan (join splitting + base-relation walking) — guessing where lowering ends and the authored expression begins. Every failure in this family was that guess going wrong:__bsl_jk_seam projections, shaped views discarded, aggregate-grain views unrecoverable, query entries losing their shaped base.This PR removes the guessing:
to_taggedstamps each leaf model's authored table expression with a marker tag inside the lowered payload (__bsl_leaf__,{"leaf": <model name>}), and recovery returns the marked subtree verbatim before any heuristic runs._collect_leaf_tablesmaps each leafSemanticTableOpto its table op (descending join wrappers'_source_join)._mark_leaf_tableswraps every structural occurrence via a node-equality rewrite — markers land wherever lowering placed the leaf: under rename projections, under pre-aggregation legs, under a query'sAggregate.What becomes first-class (catalog-round-trippable)
to_semantic_table(table.group_by(...).aggregate(...))— aggregate-grain fact models (dimensional-modeling aggregate fact tables at declared grains). Verified in-memory and through the fullxorq build → load_expr → .ls.builderdisk path with correct values.Compatibility
Tagnodes).Upstream note (xorq, untouched here)
Plain-callable replacers in ibis's
replace()must recreate nodes from_kwargsthemselves for child substitutions to propagate (Pattern/Mappingreplacers get this via_coerce_replacer; bare callables do not). The marker rewrite does this correctly — butto_tagged's pre-existingreplace_read_parquetcallable does not, and likely no-ops on nested rewrites. Worth an xorq-side look.Testing
🤖 Generated with Claude Code
Conflicts resolved (merge commit
67396ee, no history rewrite)Base gained
4a4e26f(sibling-wipe serialization fix + new tests for tagging already-aggregated join queries). The conflict was append-vs-append in the recovery test file; resolved keeping BOTH test sets — 13 tests, all passing.One honest finding from the merge: the base's new
test_join_many_aggregate_query_fails_loud_not_silently_wrongstill passes with markers. Pre-agg compilation of a taggedjoin_manyQUERY rebuilds the leaf nodes into partial-aggregate/key-bridge legs, so no structural occurrence of the authored leaf survives for the node-equality marker rewrite to find — that case remains a loud, actionable failure (tag the bare model instead; supported and tested). Markers DO rescue single-table query entries over shaped bases (test_query_entry_recovers_the_shaped_base, previously strict-xfail). Making fan-out query entries recoverable would need markers applied inside the pre-agg compiler itself — possible follow-up.