perf(schema): choose coslice witnesses during traversal, and stop early - #7
Merged
thomasmarsh merged 1 commit intoAug 16, 2026
Merged
Conversation
'materializeDecompositionCoslice' keeps, per (seed, target, direction,
ordinal), the row with the least (leg_from, leg_to). The reconstruction
did not use that: it walked every leg for every seed, expanded each leg
head's full reachable set, and built every candidate so SQL could discard
all but the minimum. On a 2,000-file corpus that is 68,683,933 leg-visits
to produce 57,968 rows.
Walking legs in ascending (leg_from, leg_to) order makes the first claim
on a target the winning one, so the answer is known without building the
rest. That in turn makes early exit sound: once every target still ahead
at an ordinal has been claimed, no remaining leg at that ordinal can
change anything, and its reachable set is never expanded.
"Still ahead at ordinal o" is exactly |{t : dist t > o}| -- a node farther
than o has a shortest path from the seed crossing ordinal o, so some leg
there reaches it. Nodes at exactly o+1 arrive as the head of their own leg
(the old finalHop); nodes beyond arrive through the head's reachable set
(the old inter).
Output is unchanged. On identical input, old build vs new:
reaches 8,537 == 8,537 md5 identical
path_leg_fwd 7,685 == 7,685
path_leg_back 6,289 == 6,289
decomposition_coslice 8,004 == 8,004 md5 identical
risk_count 1,583 == 1,583 md5 identical
column_risk 1,029 == 1,029
implied_fk 176 == 176
Schema closure phase timings:
500 files 212.7ms -> 128.7ms 1.65x
1,000 files 753.0ms -> 357.5ms 2.11x
3,000 files 918.2s -> 585.2s 1.57x
This does not make the phase sub-quadratic. The coslice output is itself
quadratic in a connected schema graph -- 9.7M path_leg_fwd rows at 3,000
files -- and no traversal beats its own output size. See thomasmarsh#6 for that
measurement; this change removes the wasted work around it, not the work
itself.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Partially addresses #6. Output is unchanged; this removes wasted work, not the
underlying quadratic.
Problem
materializeDecompositionCoslicekeeps, per(seed, target, direction, ordinal),the row with the least
(leg_from, leg_to):ROW_NUMBER() OVER (PARTITION BY seed_key, target_key, direction, leg_ordinal ORDER BY leg_from, leg_to)The reconstruction in
SchemaClosuredid not exploit that. For every seed itwalked every leg, expanded each leg head's entire reachable set, and emitted a
candidate per target — so SQL could throw all but one away. On a 2,000-file
corpus that is 68,683,933 leg-visits to produce 57,968 rows.
Change
Walk the legs at each ordinal in ascending
(leg_from, leg_to)order. Thefirst leg to claim a target is by construction the one the consumer would have
kept, so the answer is known without building the alternatives.
That makes early exit sound: once every target still ahead at an ordinal has
been claimed, no remaining leg there can change the result, and its reachable
set is never expanded.
"Still ahead at ordinal
o" is exactly|{t : dist t > o}|— a node fartherthan
ohas a shortest path from the seed crossing ordinalo, so some legthere reaches it. Nodes at exactly
o+1arrive as the head of their own leg(the old
finalHop); nodes beyond arrive via the head's reachable set (the oldinter). The count comes from a suffix scan of the distance histogram, so itis O(distinct distances), not a filter per ordinal.
Verification
Old build vs new, identical input, every table in the affected chain:
reachespath_leg_fwdpath_leg_backdecomposition_coslicerisk_countcolumn_riskimplied_fkThe md5s are order-independent digests over every column of every row, not row
counts.
cabal test all— theSchemaClosuregroup passes in full, including thegolden
path_leg_fwdcontent case and the diamond no-blow-up case. The 8failures in that run are pre-existing on
mainin this environment (the corpusloader, and seven SQL-bridge tests needing
python3, which the Haskell imagelacks).
Effect
Schema closure phase:
Whole run at 3,000 files: 1,103s -> 755s.
What this does not fix
The phase stays quadratic. The coslice output is quadratic in a connected
schema graph — 9,717,942
path_leg_fwdrows at 3,000 files — and no traversalbeats its own output size. #6 has that measurement. If the intent is for this
to scale past a few thousand objects, something has to bound the output itself
(path length, on-demand computation per requested seed, or similar), and that
is a design call rather than an optimisation.
Also tried, and not proposed
Emitting per-node counts from the closure instead of the
reachespair table,since that table is written once and read once by a
GROUP BYreducing it backto one row per node. Behaviour-preserving, but worth ~2% and it changes the
database schema, so it did not seem worth your review. Happy to send it if you
disagree.