Skip to content

perf(schema): choose coslice witnesses during traversal, and stop early - #7

Merged
thomasmarsh merged 1 commit into
thomasmarsh:mainfrom
jimmykirk:fix-coslice-output-sensitive
Aug 16, 2026
Merged

perf(schema): choose coslice witnesses during traversal, and stop early#7
thomasmarsh merged 1 commit into
thomasmarsh:mainfrom
jimmykirk:fix-coslice-output-sensitive

Conversation

@jimmykirk

Copy link
Copy Markdown
Contributor

Partially addresses #6. Output is unchanged; this removes wasted work, not the
underlying quadratic.

Problem

materializeDecompositionCoslice keeps, per (seed, target, direction, ordinal),
the row with the least (leg_from, leg_to):

ROW_NUMBER() OVER (PARTITION BY seed_key, target_key, direction, leg_ordinal
                   ORDER BY leg_from, leg_to)

The reconstruction in SchemaClosure did not exploit that. For every seed it
walked every leg, expanded each leg head's entire reachable set, and emitted a
candidate per target — so SQL could throw all but one away. On a 2,000-file
corpus that is 68,683,933 leg-visits to produce 57,968 rows.

Change

Walk the legs at each ordinal in ascending (leg_from, leg_to) order. The
first leg to claim a target is by construction the one the consumer would have
kept, so the answer is known without building the alternatives.

That makes early exit sound: once every target still ahead at an ordinal has
been claimed, no remaining leg there can change the result, and its reachable
set is never expanded.

"Still ahead at ordinal o" is exactly |{t : dist t > o}| — a node farther
than o has a shortest path from the seed crossing ordinal o, so some leg
there reaches it. Nodes at exactly o+1 arrive as the head of their own leg
(the old finalHop); nodes beyond arrive via the head's reachable set (the old
inter). The count comes from a suffix scan of the distance histogram, so it
is O(distinct distances), not a filter per ordinal.

Verification

Old build vs new, identical input, every table in the affected chain:

table base new
reaches 8,537 8,537 md5 identical
path_leg_fwd 7,685 7,685 same
path_leg_back 6,289 6,289 same
decomposition_coslice 8,004 8,004 md5 identical
risk_count 1,583 1,583 md5 identical
column_risk 1,029 1,029 same
implied_fk 176 176 same

The md5s are order-independent digests over every column of every row, not row
counts.

cabal test all — the SchemaClosure group passes in full, including the
golden path_leg_fwd content case and the diamond no-blow-up case. The 8
failures in that run are pre-existing on main in this environment (the corpus
loader, and seven SQL-bridge tests needing python3, which the Haskell image
lacks).

Effect

Schema closure phase:

files before after
500 212.7 ms 128.7 ms 1.65x
1,000 753.0 ms 357.5 ms 2.11x
3,000 918.2 s 585.2 s 1.57x

Whole run at 3,000 files: 1,103s -> 755s.

What this does not fix

The phase stays quadratic. The coslice output is quadratic in a connected
schema graph — 9,717,942 path_leg_fwd rows at 3,000 files — and no traversal
beats its own output size. #6 has that measurement. If the intent is for this
to scale past a few thousand objects, something has to bound the output itself
(path length, on-demand computation per requested seed, or similar), and that
is a design call rather than an optimisation.

Also tried, and not proposed

Emitting per-node counts from the closure instead of the reaches pair table,
since that table is written once and read once by a GROUP BY reducing it back
to one row per node. Behaviour-preserving, but worth ~2% and it changes the
database schema, so it did not seem worth your review. Happy to send it if you
disagree.

'materializeDecompositionCoslice' keeps, per (seed, target, direction,
ordinal), the row with the least (leg_from, leg_to). The reconstruction
did not use that: it walked every leg for every seed, expanded each leg
head's full reachable set, and built every candidate so SQL could discard
all but the minimum. On a 2,000-file corpus that is 68,683,933 leg-visits
to produce 57,968 rows.

Walking legs in ascending (leg_from, leg_to) order makes the first claim
on a target the winning one, so the answer is known without building the
rest. That in turn makes early exit sound: once every target still ahead
at an ordinal has been claimed, no remaining leg at that ordinal can
change anything, and its reachable set is never expanded.

"Still ahead at ordinal o" is exactly |{t : dist t > o}| -- a node farther
than o has a shortest path from the seed crossing ordinal o, so some leg
there reaches it. Nodes at exactly o+1 arrive as the head of their own leg
(the old finalHop); nodes beyond arrive through the head's reachable set
(the old inter).

Output is unchanged. On identical input, old build vs new:

    reaches                  8,537  ==  8,537   md5 identical
    path_leg_fwd             7,685  ==  7,685
    path_leg_back            6,289  ==  6,289
    decomposition_coslice    8,004  ==  8,004   md5 identical
    risk_count               1,583  ==  1,583   md5 identical
    column_risk              1,029  ==  1,029
    implied_fk                 176  ==    176

Schema closure phase timings:

      500 files   212.7ms  ->  128.7ms   1.65x
    1,000 files   753.0ms  ->  357.5ms   2.11x
    3,000 files   918.2s   ->  585.2s    1.57x

This does not make the phase sub-quadratic. The coslice output is itself
quadratic in a connected schema graph -- 9.7M path_leg_fwd rows at 3,000
files -- and no traversal beats its own output size. See thomasmarsh#6 for that
measurement; this change removes the wasted work around it, not the work
itself.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants