Skip to content

refactor phase 3a: break the core knot — the import graph is now a DAG - #298

Merged
hussainsultan merged 3 commits into
mainfrom
refactor/phase3-core-knot
Aug 18, 2026
Merged

refactor phase 3a: break the core knot — the import graph is now a DAG#298
hussainsultan merged 3 commits into
mainfrom
refactor/phase3-core-knot

Conversation

@hussainsultan

Copy link
Copy Markdown
Collaborator

Phase 3a of the SCC-driven refactor, stacked on #297. This PR takes the residual 10-module strongly connected component to zero: the intra-package import graph (function-local imports included) is now acyclic, and the ratchet test enforces that permanently. Full suite green (1703 passed, exit 0).

Guard first: SQL-snapshot corpus

Before touching the compiler's structure, ten representative query shapes (flat aggregate, filter+order+limit, derived time dimension, join_one, join_many fan-out pre-agg, mean decomposition, calc percent-of-total, three-way join, filtered join, JSON query path) now compile to golden SQL files in tests/snapshots/. Every subsequent commit in this PR — and everything in phase 3b — must keep them byte-identical; BSL_UPDATE_SNAPSHOTS=1 regenerates after an intentional change. All snapshots are unchanged by this PR.

The cuts

ops no longer imports expr, query, or format — the compiler leaves the SCC:

  • The 12 lazy from .expr import sites (fluent wrapper construction on SemanticJoinOp/SemanticIndexOp, _semantic_table) and the build_query site go through documented call-time module accessors — the same upward-dispatch pattern .chart() established. A follow-up moves those convenience methods onto the wrappers entirely; the accessors mark the debt explicitly.
  • expr.SemanticJoin.group_by/filter used to delegate to op methods that imported expr back — a ping-pong cycle. They now construct their wrappers directly.
  • _collect_struct (flavor-aware struct collection, i.e. nest lowering) moves from expr.py down into ops.py where it is consumed.
  • The SemanticUnnest isinstance check duck-types on SemanticUnnestOp instead.
  • ops imports fmt from ibis directly — the same singledispatch registry format.py registers the semantic handlers on — instead of importing the format module.

expr no longer imports query or serialization:

  • .query()/.compare_periods() and to_tagged() dispatch upward at call time. These are genuine layer-above conveniences on the expression API (like pandas' .plot), now resolved without an import-time edge.

Scanner refinement: if TYPE_CHECKING: blocks are skipped — type-only imports are not runtime edges.

Result

Tarjan over 60+ modules finds zero non-trivial SCCs. KNOWN_SCC_MEMBERS is the empty set; any future cycle fails CI. The target layering is now realizable: _xorq → primitives → ops → expr → api/query → serialization → extras → __init__, with three documented call-time upward dispatches at user-API boundaries (chart, query, serialization).

What phase 3b covers (next PR)

The physical restructure now that edges can't regress: split ops.py (~7,600 lines) into an ops/ package along the audited zone map; extract the five _to_untagged_with_* compile strategies into a compile/ subpackage; decompose the 1,089-line _to_untagged_with_preagg along its own # --- N. phase markers with an explicit PreaggContext; move pruning/lineage into compile/pruning.py. The snapshot corpus and soundness suites from this stack are the gates.

Verification

  • python3 -m pytest src/boring_semantic_layer: 1703 passed, exit 0
  • SQL snapshots: byte-identical across every commit
  • SCC ratchet: 10 → 5 → 0 in-step with the cuts; ruff check/format --check clean

🤖 Generated with Claude Code

hussainsultan and others added 3 commits August 18, 2026 19:36
Ten representative query shapes (flat agg, filters+order+limit, derived
time dim, join_one, join_many fan-out pre-agg, mean decomposition, calc
percent-of-total, three-way join, filtered join, JSON query path)
compile to golden SQL files. Any compilation-output drift during the
restructure fails loudly; BSL_UPDATE_SNAPSHOTS=1 regenerates after an
intentional change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… leaves the SCC

- The 12 lazy 'from .expr import' sites (fluent wrapper construction on
  SemanticJoinOp/SemanticIndexOp, _semantic_table) and the build_query
  site now go through documented call-time module accessors — the same
  upward-dispatch pattern as .chart(); follow-up moves those methods
  onto the wrappers entirely
- expr.SemanticJoin.group_by/filter construct their wrappers directly
  instead of ping-ponging through the op layer
- _collect_struct (flavor-aware struct collection — nest lowering)
  moves from expr.py down into ops.py where it's consumed
- The SemanticUnnest isinstance check duck-types on SemanticUnnestOp
- ops imports fmt from ibis directly (same singledispatch registry the
  format module registers handlers on) instead of importing format
- The SCC scanner now skips TYPE_CHECKING blocks: type-only imports are
  not runtime edges

SQL snapshots byte-identical; SCC: 10 -> 5 modules (expr/query/api +
serialization/{__init__,reconstruct}).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
expr's .query()/.compare_periods() and to_tagged() convenience methods
now dispatch upward at call time (the .chart() pattern) instead of
importing the query and serialization layers. That was the last cycle:
Tarjan over the full graph (lazy imports included) finds zero
non-trivial SCCs. The ratchet now enforces permanent acyclicity.

Full suite green (1703 passed, exit 0); SQL snapshots byte-identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hussainsultan
hussainsultan force-pushed the refactor/phase3-core-knot branch from e9d512b to 016622a Compare August 18, 2026 23:37
@hussainsultan
hussainsultan changed the base branch from refactor/phase2-primitives-and-errors to main August 18, 2026 23:37
@hussainsultan hussainsultan reopened this Aug 18, 2026
@hussainsultan
hussainsultan marked this pull request as ready for review August 18, 2026 23:42
@hussainsultan
hussainsultan merged commit a396044 into main Aug 18, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant