fix(schema): emit one coslice witness per ordinal instead of a cross product - #5
Merged
Conversation
…product
'cosliceClosure' emitted every shortest leg once for every target
reachable past it, on the stated assumption that this was "bounded 2x".
It is not. The row count is
|legs on shortest paths| * |targets reachable past them|
which grows with graph density, not with path length. On a 300-object
corpus it reached 29,786,641 rows for ~106k (seed, target) pairs -- 281
rows per pair.
Nothing consumed that fan-out. 'materializeDecompositionCoslice' ranks
with ROW_NUMBER() PARTITION BY (seed, target, direction, ordinal) ORDER
BY (leg_from, leg_to) and keeps rn = 1, so all but one row per ordinal
was built only to be discarded -- after a window function had sorted all
29.8M of them and joined them five ways. That is what exhausted memory.
This picks the same witness at emission time, in Haskell, using the same
total order. Output is unchanged; the intermediate no longer exists.
Measured on a fixed 150-object input, old build vs new:
path_leg_fwd 41,332 -> 10,961
decomposition_coslice 10,939 == 10,939 content md5 identical
reaches, column_risk, live_proc, dead_code, taint_paths,
schema_objects all identical
At 300 objects, where the old build could not finish at all:
path_leg_fwd 29,786,641 -> 617,296 (48x)
wall clock OOM / SIGSEGV -> 49s
The existing diamond regression test bounded the number of distinct
*targets*, which never grew -- the blow-up was in rows *per* target. The
added test asserts the invariant the materializer actually relies on:
at most one row per (seed, target, ordinal).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4.
Root cause
cosliceClosureemits, for every shortest leg, one row per target reachablepast it:
The module header describes this as "set semantics through a diamond, bounded
2x". That bound does not hold — the row count is
which grows with graph density, not with path length. On a 300-object
corpus that reached 29,786,641 rows for ~106k (seed, target) pairs, i.e.
281 rows per pair.
Why it is pure waste
Nothing consumes the fan-out.
materializeDecompositionCosliceranks withROW_NUMBER() OVER (PARTITION BY seed_key, target_key, direction, leg_ordinal ORDER BY leg_from, leg_to)and keeps
rn = 1. So all but one row per ordinal is built only to be thrownaway — after a window function has sorted all 29.8M of them and joined them
five ways (once to
schema_objectsfor the target filter, once toschema_morphisms, then four moreLEFT JOINs to decode keys). That sort andthose joins are what exhaust memory.
The change
Pick the same witness at emission time, in Haskell, using the same total order
the SQL uses.
Map.fromListWithkeyed by(target, ordinal), keeping theleast
(leg_from, leg_to). Output unchanged; the intermediate never exists.This is the same shape as the fix already applied to taint paths — the comment
on
materializeTaintPathsnotestaint_step_kindis now a Haskell BFSreconstruction for exactly this reason. The schema coslice never got the same
treatment.
Verification
Old build vs new, identical 150-object input:
path_leg_fwd(intermediate)decomposition_coslicereachescolumn_risklive_procdead_codetaint_pathsschema_objectsThe coslice comparison is an order-independent md5 over every column of every
row, not just a count.
At 300 objects, where the old build could not finish at all:
path_leg_fwdReproducible across repeated runs;
path_leg_fwdanddecomposition_coslicecounts are byte-stable run to run.
Test
The existing diamond case asserted that the number of distinct targets stays
bounded — which it always did. The blow-up was in rows per target, so nothing
measured it. The added case asserts the invariant the materializer actually
relies on: at most one row per
(seed, target, ordinal), forward and backward.Unrelated observation
While measuring this I noticed
taint_pathsrow counts vary run to run onidentical input (2,894 vs 4,012 on a 150-object corpus). That reproduces on an
unpatched build too, so it is pre-existing and independent of this change —
mentioning it only because it shows up in the table above if you run it twice.
Happy to file separately if useful.