Skip to content

fix(schema): emit one coslice witness per ordinal instead of a cross product - #5

Merged
thomasmarsh merged 1 commit into
thomasmarsh:mainfrom
jimmykirk:fix-coslice-crossproduct
Aug 16, 2026
Merged

fix(schema): emit one coslice witness per ordinal instead of a cross product#5
thomasmarsh merged 1 commit into
thomasmarsh:mainfrom
jimmykirk:fix-coslice-crossproduct

Conversation

@jimmykirk

Copy link
Copy Markdown
Contributor

Fixes #4.

Root cause

cosliceClosure emits, for every shortest leg, one row per target reachable
past it:

inter = [ [s, t, show o, lf, lt, k]
        | t <- Set.toList (reach ! lt)
        , dt > o + 1 ]

The module header describes this as "set semantics through a diamond, bounded
2x". That bound does not hold — the row count is

|legs on shortest paths| * |targets reachable past them|

which grows with graph density, not with path length. On a 300-object
corpus that reached 29,786,641 rows for ~106k (seed, target) pairs, i.e.
281 rows per pair.

Why it is pure waste

Nothing consumes the fan-out. materializeDecompositionCoslice ranks with

ROW_NUMBER() OVER (PARTITION BY seed_key, target_key, direction, leg_ordinal
                   ORDER BY leg_from, leg_to)

and keeps rn = 1. So all but one row per ordinal is built only to be thrown
away — after a window function has sorted all 29.8M of them and joined them
five ways (once to schema_objects for the target filter, once to
schema_morphisms, then four more LEFT JOINs to decode keys). That sort and
those joins are what exhaust memory.

The change

Pick the same witness at emission time, in Haskell, using the same total order
the SQL uses. Map.fromListWith keyed by (target, ordinal), keeping the
least (leg_from, leg_to). Output unchanged; the intermediate never exists.

This is the same shape as the fix already applied to taint paths — the comment
on materializeTaintPaths notes taint_step_kind is now a Haskell BFS
reconstruction for exactly this reason. The schema coslice never got the same
treatment.

Verification

Old build vs new, identical 150-object input:

table old new
path_leg_fwd (intermediate) 41,332 10,961 3.8x fewer
decomposition_coslice 10,939 10,939 content md5 identical
reaches 12,037 12,037 identical
column_risk 896 896 identical
live_proc 801 801 identical
dead_code 93 93 identical
taint_paths 2,894 2,894 identical
schema_objects 2,159 2,159 identical

The coslice comparison is an order-independent md5 over every column of every
row, not just a count.

At 300 objects, where the old build could not finish at all:

old new
path_leg_fwd 29,786,641 617,296 (48x)
outcome SIGSEGV / OOM-killed exit 0 in 49s

Reproducible across repeated runs; path_leg_fwd and decomposition_coslice
counts are byte-stable run to run.

Test

The existing diamond case asserted that the number of distinct targets stays
bounded — which it always did. The blow-up was in rows per target, so nothing
measured it. The added case asserts the invariant the materializer actually
relies on: at most one row per (seed, target, ordinal), forward and backward.

Unrelated observation

While measuring this I noticed taint_paths row counts vary run to run on
identical input (2,894 vs 4,012 on a 150-object corpus). That reproduces on an
unpatched build too, so it is pre-existing and independent of this change —
mentioning it only because it shows up in the table above if you run it twice.
Happy to file separately if useful.

…product

'cosliceClosure' emitted every shortest leg once for every target
reachable past it, on the stated assumption that this was "bounded 2x".
It is not. The row count is

    |legs on shortest paths| * |targets reachable past them|

which grows with graph density, not with path length. On a 300-object
corpus it reached 29,786,641 rows for ~106k (seed, target) pairs -- 281
rows per pair.

Nothing consumed that fan-out. 'materializeDecompositionCoslice' ranks
with ROW_NUMBER() PARTITION BY (seed, target, direction, ordinal) ORDER
BY (leg_from, leg_to) and keeps rn = 1, so all but one row per ordinal
was built only to be discarded -- after a window function had sorted all
29.8M of them and joined them five ways. That is what exhausted memory.

This picks the same witness at emission time, in Haskell, using the same
total order. Output is unchanged; the intermediate no longer exists.

Measured on a fixed 150-object input, old build vs new:

    path_leg_fwd            41,332  ->  10,961
    decomposition_coslice   10,939  ==  10,939   content md5 identical
    reaches, column_risk, live_proc, dead_code, taint_paths,
    schema_objects                          all identical

At 300 objects, where the old build could not finish at all:

    path_leg_fwd        29,786,641  ->  617,296   (48x)
    wall clock          OOM / SIGSEGV ->  49s

The existing diamond regression test bounded the number of distinct
*targets*, which never grew -- the blow-up was in rows *per* target. The
added test asserts the invariant the materializer actually relies on:
at most one row per (seed, target, ordinal).
@thomasmarsh
thomasmarsh merged commit 6d5ad97 into thomasmarsh:main Aug 16, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Analysis memory grows superlinearly in window (.srw) count; 815 windows exhaust 44 GB

2 participants