Meta: benchmark coverage unlock plan for GFQL and GFQL/Cypher

[lmeyerov]

Summary

Meta-tracker for benchmark-impacting GFQL/Cypher compiler/planner gaps.

Active Queue (as of 2026-05-10 PT)

• [FEA] GFQL bindings_table (row bindings) from chain using Yannakakis prune #880 — residual umbrella on post-rerun split set
  • [POST-RERUN] #880 residual: joined row projection bucket #1411 (joined row projection) via Add GFQL row join apply (#1411) #1426; pygraphistry-side merged, pending adapter workaround removal + fresh residual rerun
  • [POST-RERUN] #880 residual: joined row aggregation bucket #1413 (joined row aggregation; [WIP] Fix #1413 joined-row aggregation residual #1422 merged IC4/new-topics only, IC3/cross-country remains blocked on collect-list + carried-row MATCH after WITH)
  • [POST-RERUN] #880 residual: reply-author row-shaping bucket #1412 (reply-author row-shaping; [GFQL] Lock reply-author row-shaping coverage #1425 + graphistry/pyg-bench#9; DGX staged sf0.1 closeout snb-interactive-c8dee7f018 passed 2/2 GFQL probes, 0 partials, empty issue_refs)
  • [POST-RERUN] #880 residual: employment-company row-join bucket #1414 (employment-company row-join) via test: cover gfql employment company residual bucket #1424
  • [POST-RERUN] #880 residual: recursive-ancestor row-join bucket #1416 (recursive-ancestor row-join; [codex] Add #1416 recursive ancestor row-join regression #1423 merged, closeout blocked because current pyg-bench still reports message-forum as adapter_workaround partial under [FEA] GFQL bindings_table (row bindings) from chain using Yannakakis prune #880 and rerun attempts produced no usable probes)
  • [POST-RERUN] #880 residual: tag-cooccurrence join-aggregation bucket #1415 (tag-cooccurrence join-aggregation; draft PR [codex] Tie #1415 to IC6 tag-cooccurrence coverage #1427 active)
• GFQL row pipeline: cuDF host-bridge gap in order_by + test placement debt for compute/gfql/row/ (follow-ups from #1362/#1359) #1373 — row-pipeline host-bridge/test follow-up via fix(gfql/row): close #1373 cuDF host bridge + row test follow-ups #1404 (merged 2026-05-10)
• [FEA] Cypher binder: enable strict-name-resolution to centralize alias-scope enforcement #1357 — strict-name-resolution rollout completion via GFQL/Cypher: strict runtime binder default + baseline retirement (#1357) #1408
• Meta: Prioritized direct-Cypher wrong-row bugfix tranches (92 cases) #1353 tranches A/B/C complete
  • [META-split] Direct-Cypher wrong-rows tranche A: comparator + ORDER BY semantics #1405 comparator + ORDER BY (via Fix GFQL structural null equality semantics for list/map comparisons (#1405) #1410)
  • [META-split] Direct-Cypher wrong-rows tranche B: temporal comparison/canonicalization #1406 temporal comparison/canonicalization (via fix(cypher): temporal historical canonicalization/comparison tranche #1409)
  • [META-split] Direct-Cypher wrong-rows tranche C: tri-valued list/map/null semantics #1407 tri-valued list/map/null (via fix(gfql): tri-valued list/map/null semantics tranche (#1407) #1417)
• Meta: Prioritized direct-Cypher wrong-row bugfix tranches (92 cases) #1353 tranche D (string/type-conversion cleanup) to file/dispatch

Queued Cleanup Wave (after gates)

• [META] GFQL deletion wave: remove 2K+ LOC of obsolete compatibility paths #1419 — 2K+ LOC GFQL deletion wave (gated)
  • [DEL] D1 remove obsolete row-join fallback branches after #880 post-rerun fixes #1418 D1 row-join fallback deletion
  • [DEL] D2 remove pre-strict binder compatibility guards after #1357 #1420 D2 pre-strict binder guard deletion
  • [DEL] D3 remove stale reentry failfast scaffolding #1421 D3 reentry failfast scaffolding deletion

Dispatch Guidance

Active / in-flight

• [POST-RERUN] #880 residual: tag-cooccurrence join-aggregation bucket #1415 via draft PR [codex] Tie #1415 to IC6 tag-cooccurrence coverage #1427: tag-cooccurrence join-aggregation coverage/fix lane.
• [POST-RERUN] #880 residual: joined row aggregation bucket #1413 follow-up: IC3/cross-country remains after [WIP] Fix #1413 joined-row aggregation residual #1422 and likely needs a compiler slice for collect-list + carried rows + NOT friendCity IN cities + multiple post-reentry WITH stages.

Merged but pending final residual closeout

• [POST-RERUN] #880 residual: joined row projection bucket #1411 via Add GFQL row join apply (#1411) #1426: pygraphistry-side joined row projection support merged; run closeout lane when the remaining adapter workaround can be inspected/removed, then rerun residuals.
• [POST-RERUN] #880 residual: recursive-ancestor row-join bucket #1416 via [codex] Add #1416 recursive ancestor row-join regression #1423: verification is blocked; investigate why pyg-bench still marks message-forum GFQL as adapter_workaround partial and why rerun/probe artifacts are unavailable.

Completed in active wave

• [POST-RERUN] #880 residual: reply-author row-shaping bucket #1412 via [GFQL] Lock reply-author row-shaping coverage #1425 + graphistry/pyg-bench#9
• [POST-RERUN] #880 residual: employment-company row-join bucket #1414 via test: cover gfql employment company residual bucket #1424

Related

• Meta: GFQL compiler/planner post-M4 execution DAG (active wave) #1259 execution DAG meta
• [FEA] GFQL bindings_table (row bindings) from chain using Yannakakis prune #880 benchmark residual umbrella

[lmeyerov]

New measured benchmark state from pyg-bench as of March 31, 2026:

• official SNB query floor measured: 7 (interactive-short-1/2/3/4/5/6 + interactive-complex-2)
• GFQL: ok=2, supported_with_workaround=5
• Cypher: ok=3, blocked=3, semantic_mismatch=1

New issue from that floor:

• #994 direct Cypher undirected peer binding mismatch on incoming edges
  • blocks interactive-short-3 correctness even though the query now executes

Current local next queue from the translation analyzer:

1. interactive-complex-8
2. interactive-short-7
3. interactive-complex-13
4. interactive-complex-4

[lmeyerov]

Priority recommendation from the current measured SNB benchmark floor (8 official queries measured on March 31, 2026):

Coverage unlock order:

1. #981 + #880 together
   • biggest unlock by far: each touches 13 official SNB queries
   • current measured manifestations include:
     • interactive-complex-2 blocked on direct Cypher multi-binding projection
     • interactive-complex-8 semantic corruption on direct Cypher row lowering
     • multiple GFQL paths still relying on local join workarounds instead of direct row materialization
2. #973
   • next official-query unlock for connected varlen patterns
   • currently blocks the official interactive-short-2 / interactive-short-6 family
3. #982
   • narrower than the row-binding work, but still needed to remove the current direct GFQL edge-projection workaround on interactive-short-3

Run in parallel as a correctness fix:

• #994
  • narrower query impact than the three unlocks above, but it is a wrong-answer bug in a query shape that already executes
  • current measured manifestation: interactive-short-3 returns the seed node as the peer alias on incoming undirected KNOWS edges

So the benchmark recommendation is:

• main unlock track: #981/#880 -> #973 -> #982
• correctness fast-path: #994 in parallel

[lmeyerov]

New measured benchmark state from pyg-bench as of March 31, 2026 after adding interactive-short-7:

• official SNB query floor measured: 9
  • interactive-short-1/2/3/4/5/6/7
  • interactive-complex-2/8
• GFQL: ok=2, supported_with_workaround=7
• Cypher: ok=3, blocked=4, semantic_mismatch=2

Important change: interactive-short-7 is the first measured query that clearly opens a new post-row-shaping feature lane.

Measured result:

• GFQL: workaround-backed partial
• direct Cypher: primitive_gap via new issue #996
  • MATCH ... OPTIONAL MATCH ... RETURN CASE ...
  • current error: Only node-only pre-binding MATCH clauses are supported before the final connected MATCH in this phase

Updated benchmark priority recommendation:

1. #981 + #880
2. #973
3. #982
4. #996 as the first clearly new feature-lowering lane beyond row shaping
5. #994 in parallel as the narrower wrong-answer bug

Current next translation queue:

1. interactive-complex-13
2. interactive-complex-4
3. interactive-complex-7
4. interactive-complex-11

[lmeyerov]

Updated benchmark coverage status from pyg-bench on March 31, 2026 using explicit pygraphistry master snapshot 7095102df:

• SNB Interactive official query texts inventoried: 29
• Measured today: 12
• Current measured floor: interactive-short-1/2/3/4/5/6/7 plus interactive-complex-2/4/8/11/13
• Dataset-backed GFQL status on the measured floor: ok=2, supported_with_workaround=10
• Dataset-backed Cypher status on the measured floor: ok=3, blocked=7, semantic_mismatch=2

Newly added measured queries in this pass:

• interactive-complex-4 / new-topics: GFQL workaround passes; direct Cypher blocked by #981
• interactive-complex-11 / job-referral: GFQL workaround passes; direct Cypher blocked by #981

Updated next translation queue from the analyzer:

1. interactive-complex-3
2. interactive-complex-7
3. interactive-complex-5
4. interactive-complex-1
5. interactive-complex-10

Priority order still holds: #981/#880, then #973, then #982, with #996, #997, and #994 as the next measured side lanes.

[lmeyerov]

Updated measured benchmark state from pyg-bench on March 31, 2026 after adding interactive-complex-3 and reproducing the expanded floor on dgx-spark CPU:

• SNB Interactive official query texts inventoried: 29
• measured today: 14
• current measured floor:
  • interactive-short-1/2/3/4/5/6/7
  • interactive-complex-2/3/4/7/8/11/13
• dataset-backed GFQL status on the measured floor: ok=2, supported_with_workaround=12
• dataset-backed Cypher status on the measured floor: ok=3, blocked=9, semantic_mismatch=2
• latest remote reproduction: dgx-spark CPU run snb-interactive-358480dd77
  • status mix: ok=5, partial=12, planner_gap=3, primitive_gap=4, result_mismatch=2, parser_gap=2

New measured query in this pass:

• interactive-complex-3 / cross-country-messages
  • GFQL: workaround-backed partial
  • direct Cypher: new issue #999
    • multi-stage MATCH ... WITH ... MATCH ... WITH ... MATCH read-query reentry
    • current error: Cypher WITH after post-WITH MATCH is not yet supported in the current GFQL Cypher compiler

Updated benchmark unlock order from the current floor:

1. #981 + #880
2. #999
3. #973
4. #982
5. #996
6. #997
7. #998
8. #994 in parallel as the narrower wrong-answer fix

Updated next translation queue from the analyzer:

1. interactive-complex-1
2. interactive-complex-10
3. interactive-complex-12
4. interactive-complex-5
5. interactive-complex-9
6. interactive-complex-6

[lmeyerov]

Benchmark update from pyg-bench on 2026-03-31:

• Measured SNB Interactive floor is now 15/29 official queries.
• Latest remote artifact on dgx-spark CPU: results/runs/dgx-spark-snb-interactive-master-7095102df-v11/
• Current remote status mix on the 15-query floor: ok=5, partial=13, planner_gap=3, primitive_gap=5, result_mismatch=2, parser_gap=2.

New measured blocker from interactive-complex-9 (Recent messages by friends or friends of friends):

• direct Cypher is blocked by WITH collect(...) -> UNWIND -> MATCH continuation after graph aliases are introduced
• tracked separately in #1000
• GFQL still covers the query via a 1..2-hop friend plus local message-join workaround under #880

The current measured unlock order is now:

1. #981 + #880
2. #1000 + #999
3. #973
4. #982
5. #996 / #997 / #998
6. #994

[lmeyerov]

Another benchmark update from pyg-bench on 2026-03-31:

• Measured SNB Interactive floor is now 16/29 official queries.
• Latest remote artifact on dgx-spark CPU: results/runs/dgx-spark-snb-interactive-master-7095102df-v12/
• Current remote status mix on the 16-query floor: ok=5, partial=14, planner_gap=3, primitive_gap=5, result_mismatch=2, parser_gap=3.

New measured query added:

• interactive-complex-6 (Tag co-occurrence)
  • GFQL: partial via 1..2-hop friend plus local tag co-occurrence aggregation workaround under #880
  • Cypher: parser_gap under #1000

Analyzer state after adding complex-6 and complex-9:

• #1000 now affects 6 official queries with 2 already measured
• next lowest-novelty queue is now interactive-complex-12, interactive-complex-5, interactive-complex-1, interactive-complex-10

[lmeyerov]

Benchmark update from pyg-bench on 2026-03-31:

• Measured SNB Interactive floor is now 17/29 official queries.
• Latest remote artifact on dgx-spark CPU: results/runs/dgx-spark-snb-interactive-master-7095102df-v13/
• Current remote status mix on the 17-query floor: ok=5, partial=15, planner_gap=3, primitive_gap=5, result_mismatch=2, parser_gap=4.

New measured query added:

• interactive-complex-12 (Expert search)
  • GFQL: partial via direct-friend plus local tag-class-expanded reply aggregation workaround under #880
  • Cypher: parser_gap under #983

Important correction:

• #984 improved the earlier REPLY_OF*0.. manifestation, but #983 remains active for relationship-alternation zero-hop forms such as [:HAS_TYPE|IS_SUBCLASS_OF*0..].

Next queue from the analyzer is now:

• interactive-complex-5
• interactive-complex-1
• interactive-complex-10
• interactive-complex-14

[lmeyerov]

Benchmark coverage update from pyg-bench on March 31, 2026:

• official SNB Interactive query inventory: 29
• measured floor: 18
• backend status on the measured floor:
  • GFQL: ok=2, supported_with_workaround=16
  • direct Cypher: ok=3, blocked=13, semantic_mismatch=2
• latest remote conformance artifact: dgx-spark-snb-interactive-master-7095102df-v14
  • status mix: ok=5, partial=16, planner_gap=3, primitive_gap=6, result_mismatch=2, parser_gap=4

New measured query this pass:

• interactive-complex-5 / new-groups
  • GFQL: partial via friend-membership + forum-post-count workaround
  • direct Cypher: blocked by existing #999 lane, no new issue family needed

Current read-side next queue from the translation analyzer:

1. interactive-complex-1
2. interactive-complex-10
3. interactive-complex-14

[lmeyerov]

Benchmark coverage update from pyg-bench on March 31, 2026:

• official SNB Interactive query inventory: 29
• measured floor: 19
• backend status on the measured floor:
  • GFQL: ok=2, supported_with_workaround=17
  • direct Cypher: ok=3, blocked=14, semantic_mismatch=2
• latest remote conformance artifact: dgx-spark-snb-interactive-master-7095102df-v15
  • status mix: ok=5, partial=17, planner_gap=3, primitive_gap=6, result_mismatch=2, parser_gap=5

New measured query this pass:

• interactive-complex-1 / named-shortest-friends
  • GFQL: partial via bounded-shortest-path + profile-history join workaround
  • direct Cypher: blocked by confirmed #997 shortestPath(...) parser/front-end lane

Current read-side next queue from the translation analyzer:

1. interactive-complex-10
2. interactive-complex-14

[lmeyerov]

Benchmark status update from pyg-bench after rerunning the measured #1000 lane against isolated ambient graphistry v0.53.11.

Artifact:

• results/runs/local-snb-interactive-v05311-rerun/

Outcome split:

• interactive-complex-9 / recent-network-messages now clears the old WITH collect(distinct friend) -> UNWIND -> MATCH blocker and fails later at #981
• interactive-complex-6 / tag-cooccurrence still reproduces #1000, but on a narrower carried-scalar shape:
  • WITH knownTagId, collect(distinct friend) AS friends
  • UNWIND friends AS f
  • MATCH ...

Practical priority implication for benchmark coverage:

1. #981/#880 stay the main unlocks
2. #999 remains the broader MATCH/WITH/MATCH continuation family
3. #1000 is now a narrower residual benchmark blocker, not a blanket blocker for every measured WITH -> UNWIND -> MATCH query

[lmeyerov]

Benchmark-driven priority update from pyg-bench as of 2026-03-31.

Measured state:

• DGX CPU sf1 core slice is still badly behind current baselines:
  • PyGraphistry GFQL: 13037.569 ms
  • PyGraphistry Cypher: 6081.106 ms
  • Kuzu: 38.894 ms
  • Neo4j: 342.673 ms
• DGX GPU now has a completed SNB-shaped subset, but it is still conformance-limited:
  • single-query GFQL seed-lookup: 221.807 ms
  • core 3-query GFQL total: 612.941 ms
  • core 3-query Cypher total: 565.969 ms
  • still not apples-to-apples because one-hop-expand mismatches and expand-order-limit still lands on #973

What the language/runtime team should prioritize first:

1. #880 + #981
   • not just for coverage; the current benchmark path is forced into multi-pass local row joins and that is killing both correctness and performance.
2. #994
   • wrong-answer bug on interactive-short-3 / one-hop-expand; until this is fixed, the core comparison slice is semantically dirty.
3. #973
   • still blocks official interactive-short-2 / expand-order-limit on the direct Cypher path.
4. #982
   • still forces the GFQL side into the edge-row workaround path for interactive-short-3.

Important nuance: not all current slowness is a PyGraphistry issue.

• pyg-bench is still doing too much CPU-side fixture prep before queries.
• We fixed one harness bug where narrow GPU runs still instantiated unrelated probe specs.
• The remaining big bench-side task is reducing CPU-heavy fixture prep and keeping GPU slices narrow until semantics are clean.

So the short version is:

• for PyGraphistry, prioritize #880/#981, then #994, then #973, then #982
• for pyg-bench, we will keep reducing adapter/harness overhead so runtime numbers are less polluted by bench-side workarounds

[lmeyerov]

Warmup-aware DGX sf1 core slice update from pyg-bench.

Methodology:

• load/setup split from query timing
• untimed warmup rounds = 1
• PyGraphistry rows remain official-query-grounded custom adapter
• Kuzu and Neo4j rows remain custom reduced baseline

Current steady-state query-only numbers on the reduced 3-intent slice:

• PyGraphistry CPU: 9350.831 ms
  • direct query: 2252.492 ms
  • result materialization: 0.193 ms
  • adapter overhead: 7098.146 ms
• PyGraphistry GPU: 530.804 ms
  • direct query: 178.301 ms
  • result materialization: 4.293 ms
  • adapter overhead: 348.209 ms
• Kuzu CPU: 32.996 ms
• Neo4j CPU: 41.937 ms

Important read:

• GPU is materially better than PyGraphistry CPU on this slice.
• The current PyGraphistry timed path is still dominated by workaround cost, not direct engine execution.
• On the current 3-intent slice, the only direct GFQL timing contribution is still the reduced expand-order-limit lane.
• The core blockers are still product-side: #880, #981, #982, plus correctness gap #994. The benchmark harness is cleaner now, so the remaining gap is harder to dismiss as measurement noise.

Latest artifacts in pyg-bench:

• CPU: results/runs/dgx-spark-snb-interactive-core-sf1-timing-v5/
• GPU: results/runs/dgx-spark-snb-interactive-core-sf1-gpu-timing-v9/
• Kuzu: results/runs/dgx-spark-kuzu-snb-conformance-sf1-timing-v4/
• Neo4j: results/runs/dgx-spark-neo4j-snb-conformance-sf1-timing-v4/

[lmeyerov]

Update: v0.53.15 shipped (#994, #997, #998)

PR #1009 merged and tagged v0.53.15.

Resolved

• Cypher undirected MATCH can bind the seed node as the peer alias on incoming edges #994 (correctness): Undirected MATCH wrong-answer fix — orient_edges now correctly doubles undirected edges in both directions for connected bindings.
• Cypher parser rejects shortestPath((a)-[:REL*]-(b)) official shortest-path syntax #997 (parser): shortestPath() / allShortestPaths() now parse and fail-fast with clear validation error. Execution tracked in feat(cypher): implement shortestPath execution with igraph/cugraph backends #1010.
• Cypher parser rejects boolean pattern existence expressions in RETURN #998 (parser): Pattern existence expressions (not((a)-[:R]-(b)), exists { ... }) now detected and fail-fast with clear validation error.

Test coverage added

• 14 new tests including undirected edge cases (incoming, outgoing, bidirectional, multi-edge, star, self-loop), shortestPath/allShortestPaths fail-fast, pattern existence fail-fast (RETURN + WHERE), and historic regression for parameterized LIMIT in reentry.
• GPU validated on dgx-spark (pandas + cudf).

Impact on measured floor

• interactive-short-3 (Cypher undirected MATCH can bind the seed node as the peer alias on incoming edges #994) should now return correct rows on direct Cypher path
• interactive-complex-13 (Cypher parser rejects shortestPath((a)-[:REL*]-(b)) official shortest-path syntax #997) now fails with clear message instead of generic syntax error
• interactive-complex-7 (Cypher parser rejects boolean pattern existence expressions in RETURN #998) now fails with clear message instead of generic syntax error

Remaining top blockers (unchanged)

1. Direct Cypher row lowering rejects scalar multi-alias projections from a single MATCH #981/[FEA] GFQL bindings_table (row bindings) from chain using Yannakakis prune #880 — multi-binding row materialization (13 queries)
2. Cypher multihop: expand direct variable-length support beyond endpoint-only patterns #973 — connected-pattern variable-length lowering (4 queries) — partially addressed by feat(cypher): support variable-length rels in connected patterns (#973) #995
3. AST row select cannot reference edge alias properties after traversal #982 — native AST edge alias projection (5 queries)
4. feat(cypher): implement shortestPath execution with igraph/cugraph backends #1010 — shortestPath execution (blocked by feat(cypher): support comma-separated MATCH patterns (cartesian product) #990 path variable support)