diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4b95768bc2..43a5e955b9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1193,7 +1193,7 @@ jobs: cat build/gfql-coverage-audit/gfql-coverage-audit.md >> "$GITHUB_STEP_SUMMARY" - name: Upload GFQL coverage audit - if: ${{ matrix.python-version == '3.12' }} + if: ${{ always() && matrix.python-version == '3.12' }} uses: actions/upload-artifact@v4 with: name: gfql-coverage-audit-py3.12 diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a1ad99e9c..5282c1f35d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,7 +12,6 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine ` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator). ### Added -- **GFQL Cypher row-pipeline single-alias predicate pushdown**: Planner-generated `rows(alias_prefilters=...)` hints pre-filter eligible node and single-hop edge alias frames before binding joins on pandas/cuDF while retaining the original post-join filters for exact semantics and engine-safe fallback. - **GFQL viz-filter-pipeline acceptance suite + regression benchmark (viz-filter L3)**: `test_viz_pipeline_conformance.py` — curated full-panel pipelines (node+edge filters, exclusion-dominates composition, `(pred OR IS NULL)` keep-null leaves, both EXISTS prune-isolated flavors, searchAny composition, deterministic paging), graph-state prune shapes with exact node+edge pins, a 40-seed panel-state fuzzer with an independent plain-pandas second oracle, and a case/regex/unicode trick matrix (ß/İ full-case-mapping pins, metachar literal-vs-regex, null cells, Categorical) — all parity-or-NIE across pandas/cuDF/polars/polars-gpu. `benchmarks/gfql/viz_filter_pipeline.py` — six streamgl-viz panel scenarios (filters, keep-self GRAPH prune, EXISTS prune, node/edge search, combined) at 100K/1M/10M with native-frame-per-engine fairness, an NIE-tolerant matrix, and JSON receipts (first receipt: 100k × 4 engines, everything within the ~350ms interactive reference except pandas combined). Documented findings from the first runs: the same-path WHERE route dedupes parallel edges (diverges from the panel algebra's edge multiplicity — pinned + tracked), and edge-alias searchAny declines on polars (tracked). - **GFQL Cypher `searchAny(entity, term[, opts])` cross-column search predicate + `g.search_nodes()`/`g.search_edges()` (viz-filter L2, native on all four engines)**: True where ANY of the entity's columns matches the term — the streamgl-viz inspector's table-search semantics as a composable WHERE predicate: OR across columns; case-insensitive substring by DEFAULT (case-folded, never regex — the common call avoids every engine regex limit); regex opt-in obeying the same per-engine decline rules as `=~`; dtype gate AS SEMANTICS (string columns always; integer columns iff the term is a numeric literal, per the inspector's `/^[0-9.-]+$/` gate; floats/dates/booleans reachable via the explicit `columns:` list). Options map `{caseSensitive, regex, columns}` is strict-validated (unknown keys error listing the valid ones); unbound aliases and missing explicit columns error clearly; null cells never match. Lowered like the pattern-predicate markers (a `search_any` row op + fresh marker column), so it composes through AND/OR/NOT and different node/edge terms coexist in one pipeline. Per-column matching reuses the parity-hardened `Contains` predicate on pandas/cuDF and a lowercase-fold/any_horizontal lowering on polars; oracle-pinned + 4-engine parity-or-NIE conformance cases. Python twins `g.search_nodes(term, columns=, case_sensitive=, regex=)` / `g.search_edges(...)` filter their own table and return a Plottable (polars-frame twins decline honestly for now — use the cypher op). Honest declines (NIE, use engine='pandas'): edge-alias searchAny on polars, and explicit columns beyond string/int/bool dtypes on polars AND cuDF (incl. floats: repr diverges across engines — dgx-probed). - **GFQL Cypher `EXISTS { }` pattern-existence subqueries (openCypher-standard), native on all four engines**: `WHERE EXISTS { (n)-[:R]->() }` and `WHERE NOT EXISTS { (n)--() }` now parse and run — the declarative prune-isolated building blocks for the streamgl-viz filter pipeline. An `EXISTS` body reuses the existing pattern-predicate lowering wholesale (`semi_apply_mark` / `anti_semi_apply` row ops), so pandas/cuDF worked immediately; the polars engine gains NATIVE lowerings for the semi-apply family (correlated key sets computed by the polars chain executor's named-flag columns; order-preserving `is_in` joins) plus `rows(binding_ops=...)` for the single-entity row table — previously all honest-NIE. Aliases introduced inside the braces are existentially scoped (`EXISTS { (n)--(m) }` allowed with `m` unbound outside — bare pattern predicates keep the conservative guard), inline property maps work, and the one supported inner `WHERE` form is endpoint inequality — `EXISTS { (n)--(m) WHERE m <> n }`, the drop-self-loop prune-isolated flavor (pandas/cuDF filter the correlated bindings; polars excludes self-loop edges, which is exactly the `m <> n` witness). Both prune flavors are oracle-pinned in the conformance matrix on a self-loop discriminator graph, 4-engine parity-or-NIE. Honest declines with clear errors: `EXISTS` in RETURN/WITH projections, general inner `WHERE`, multi-pattern bodies, full `MATCH..RETURN` subquery bodies, multi-alias correlation on polars. @@ -31,6 +30,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm - **GFQL native Polars engine — more cypher row coverage (`toFloat`, `collect`/`collect(DISTINCT)`, `WHERE … IN`)**: three surfaces that previously raised `NotImplementedError` on `engine='polars'` now run natively, parity-validated vs the pandas oracle across all four engines (and honest-NIE where pandas can't be matched). **`toFloat(x)`** lowers int/uint/bool/float → `Float64` (NaN preserved — float64 has no separate null sentinel, unlike `toInteger`); a non-numeric String declines (NIE) because pandas `astype(float)` *raises* rather than null-on-failure. **`collect(x)` / `collect(DISTINCT x)`** aggregations complete the native `group_by` surface (every other agg was already native): drop nulls, preserve within-group first-occurrence order (`collect` keeps dups; `DISTINCT` dedups keep-first), all-null group → `[]`. **`where_rows`/`WHERE … IN [list]`** membership lowers to `is_in` (a null cell is excluded per openCypher 3VL). No change to any already-native path. - **GFQL Polars-CPU streaming collect (opt-in, large traversals)**: `GFQL_POLARS_CPU_STREAMING=1` runs the polars-CPU lazy collects (`hop`/`chain`) on the polars **streaming** executor instead of the default in-memory collect. Benchmarked ~1.04–1.11× faster on big multi-hop traversals (10M nodes / 80M edges: 20.0→18.0 s) and parity-identical, but ~0.86× (slower) on small/interactive sizes (streaming overhead) — so it is **opt-in, default off** (no change to default behavior). Use for large batch traversals where CPU is the target. - **GFQL Cypher `count(*)` short-circuit — O(1) instead of O(N) materialize**: a lone `RETURN count(*)` over a single node or edge pattern (`MATCH (n) RETURN count(*)`, `MATCH ()-[r]->() RETURN count(*)`) previously materialized the entire matched frame and ran a constant-key `group_by` just to count its rows. The lowering now emits a new `count_table` row op that reads the scanned table's height directly (or, when the pattern applies a filter, sums the boolean alias-mask column) in a single reduction — no full-frame copy, no group_by. Applies only to the provably-equivalent shapes: exactly one non-DISTINCT `count(*)`, no group keys / post-aggregate exprs / row-level `WHERE` / `UNWIND` / paging / multi-relationship binding, and either a pure node scan (`relationship_count == 0`) or a single relationship counted on its edge alias — every other shape (node-alias-over-relationship, multi-hop paths, `count(DISTINCT …)`, grouped counts) falls through to the general aggregate path unchanged. Engine-polymorphic across pandas/cuDF/polars/polars-gpu; differential parity verified on all four engines (`count_all_nodes`/`count_all_edges` cypher conformance cases + a `count_table` row-op subject case, chain and DAG surfaces). No change to any result value — only the execution path. +- **GFQL Cypher aggregate fast paths — single-hop grouped aggregate + two-hop connected count**: grouped-aggregate `RETURN`s over the common OLAP shapes now execute on dedicated fast paths instead of the general bindings-materialize-then-group route: single-hop `MATCH (n) RETURN , count(n)/sum(n.x)/avg(n.x) [ORDER BY … LIMIT …]` and two-hop connected `MATCH (a)-[]->(b)-[]->(c) RETURN count(*)`/`count(a)`. Byte-identical to the general path on pandas/cuDF/polars/polars-gpu (differential parity vs the hand-derived openCypher oracle), with openCypher ordering honored — NULL sorts as the largest value, so a grouped `ORDER BY ASC LIMIT k` places the NULL-key bucket last (this fixed a polars nulls-first divergence). The string-Cypher compile cache keys on the resolved node dtypes so a plan compiled under one engine's dtype view is never reused for another on the same graph; the cache stores data-independent plans only (no cross-call result staleness under in-place graph mutation). Unsupported grouped shapes (e.g. `min`/`max` over a grouped bindings key) raise an honest validation error rather than silently taking a wrong path. - **GFQL `engine='polars'`/`'polars-gpu'` can now run off-engine analytic `call()`s (umap/hypergraph/compute_cugraph/…) — `call_mode='auto'` (default)**: a GFQL `call()` that invokes a Plottable-method analytic with **no native polars implementation** (and never will — `umap`, `hypergraph`, `compute_cugraph`, `compute_igraph`, `layout_*`, `collapse`, `get_topological_levels`, …) previously raised `NotImplementedError` under a polars engine, forcing users off polars for the whole pipeline. It now runs as a **mode-gated, warned modality switch**: `call_mode='auto'` (default) bridges the graph off-engine (`polars`→pandas, `polars-gpu`→cuDF **on-device**), runs the analytic, and coerces the result back to polars **losslessly via Arrow**, warning once per (process, function). `call_mode='strict'` keeps the honest `NotImplementedError` decline (for benchmark integrity / a hard memory ceiling). `polars-gpu` is **GPU-or-error**: it bridges to cuDF and declines rather than silently dropping a GPU analytic to host pandas. This is **deliberately narrower than CHAIN traversal/filter/row ops**, which stay parity-or-NIE (a bridge there would hide a missing impl and cheat a benchmark) — the split is mechanical (`is_row_pipeline_call`), and the chain and DAG surfaces bridge consistently. Configurable via `graphistry.compute.gfql.lazy.set_call_mode('auto'|'strict')` (Python) or `GFQL_POLARS_CALL_MODE` (env), resolving Python override > env > default('auto'), read live. Verified on dgx: `compute_cugraph` PageRank is byte-parity with the pandas oracle under both `polars` and `polars-gpu`. ### Changed @@ -42,9 +42,8 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm - **GFQL Cypher connected-pattern (q5–q7) planning + execution on all engines**: completes the `rows(binding_ops)` follow-on promised above — multi-alias *connected* `MATCH` patterns that filter and join across shared endpoints (`MATCH (a)-[]->(b)<-[]-(c) WHERE … RETURN …`, star/two-star shapes) now plan and execute natively on pandas, cuDF, `engine='polars'`, and `engine='polars-gpu'` (previously an honest `NotImplementedError` on polars). The connected-join planner pushes eligible single-alias endpoint predicates into per-frame `filter_dict` prefilters when the column dtype makes the pushdown exact, and keeps them as a post-join `where_rows` residual otherwise (nullable/decimal/object/predicate/`AllOf`/`toLower` shapes), so results stay byte-identical to the full-join semantics on every engine. The two-star executor adds grouped-count and bindings-row fast paths with a per-execution result cache (no cross-call staleness under in-place graph mutation) and openCypher-correct ordering — NULL sorts as the largest value (`ORDER BY … ASC` puts nulls last), and `count(a)`/`count(DISTINCT a)`/`count(*)` follow Cypher multiplicity. Differential parity vs the hand-derived openCypher oracle across all four engines (connected-join conformance corpus). ### Fixed -- **GFQL seeded indexes now engage for small Polars/cuDF frontiers**: NaN-free native Polars frames retain object identity during normalization, preserving resident CSR validity, while a tunable absolute floor prevents the fractional cost gate from routing tiny seeded hops over low-cardinality slices to a full scan. Scan/index result parity is unchanged; the default floor is a routing heuristic whose engine/data crossover remains subject to measurement. - **GFQL Cypher `GRAPH { }` residual predicates now fail safely or apply as graph masks**: `GRAPH { MATCH ... WHERE ... }` no longer silently drops predicates that the graph-state path cannot apply. Safe one-node/one-edge residual filters, including disjunctions and `searchAny(...)`, are applied before graph-state matching; unsupported pattern-predicate, multi-alias, and Polars graph-residual cases now raise clear validation errors instead of returning an over-broad subgraph. -- **GFQL Cypher `=~` / scalar-fn cross-engine hardening (review wave, dgx-verified)**: (1) composed `=~` (`WHERE … =~ … OR …`, `RETURN`-expression position) now works on `engine='cudf'` — the series evaluator used raw `.str.fullmatch`, which cuDF lacks, and the resulting `AttributeError` was masked as "unsupported predicate op" (it now routes through the `Fullmatch` predicate's engine workarounds, and honest `NotImplementedError` declines pass through instead of being re-labeled); (2) the cuDF fullmatch emulation anchors alternations as a whole (`^(ab|cd)$` — bare `^ab|cd$` silently matched `'abXXX'`); (3) the cuDF case-insensitive `(?i)` lowercase-folding workaround now declines the fold-unsound shapes — uppercase escape classes (`.lower()` turns `\D` into `\d`, silently inverting the predicate), case-crossing character ranges (`(?i)[A-z]` silently narrowed; `[X-b]` folded to an invalid range), and non-ASCII patterns — while lowercase escapes (`\d`, `\.`) keep folding as before; lookaround, backreferences, and named-group refs decline up front (libcudf rejects them at kernel-compile time); (4) polars `Match`/`Fullmatch` lowering applies the same Rust-regex guard as `Contains` (lookaround/backrefs decline instead of a non-NIE `ComputeError` at collect); (5) `toLower`/`toUpper` on a non-string column decline like neo4j's type error instead of broadcasting the stringified Series repr (pandas/cuDF) or raising a non-NIE `SchemaError` (polars-gpu); (6) polars `floor`/`ceil`/`round` cast to `Float64` so integer columns return Float like neo4j and the pandas engine; (7) invalid regex patterns on the composed path raise a clear "invalid regex pattern" error instead of "unsupported predicate op". +- **GFQL Cypher `=~` / scalar-fn cross-engine hardening (review wave, dgx-verified)**: (1) composed `=~` (`WHERE … =~ … OR …`, `RETURN`-expression position) now works on `engine='cudf'` — the series evaluator used raw `.str.fullmatch`, which cuDF lacks, and the resulting `AttributeError` was masked as "unsupported predicate op" (it now routes through the `Fullmatch` predicate's engine workarounds, and honest `NotImplementedError` declines pass through instead of being re-labeled); (2) the cuDF fullmatch emulation anchors alternations as a whole (`^(ab|cd)$` — bare `^ab|cd$` silently matched `'abXXX'`); (3) the cuDF case-insensitive `(?i)` lowercase-folding workaround now declines the fold-unsound shapes — uppercase escape classes (`.lower()` turns `\D` into `\d`, silently inverting the predicate), case-crossing character ranges (`(?i)[A-z]` silently narrowed; `[X-b]` folded to an invalid range), and non-ASCII patterns — while lowercase escapes (`\d`, `\.`) keep folding as before; lookaround, backreferences, and named-group refs decline up front (libcudf rejects them at kernel-compile time); (4) polars `Match`/`Fullmatch` lowering applies the same Rust-regex guard as `Contains` (lookaround/backrefs decline instead of a non-NIE `ComputeError` at collect); (5) `toLower`/`toUpper` on a non-string column decline with a Cypher-standard type error instead of broadcasting the stringified Series repr (pandas/cuDF) or raising a non-NIE `SchemaError` (polars-gpu); (6) polars `floor`/`ceil`/`round` cast to `Float64` so integer columns return Float per Cypher semantics, matching the pandas engine; (7) invalid regex patterns on the composed path raise a clear "invalid regex pattern" error instead of "unsupported predicate op". - **GFQL Cypher `round()` hardening (review wave, dgx-verified)**: (1) the `polars` extra's floor is now `polars>=1.29` — `Expr.round(mode=)` shipped in py-1.29.0 (pola-rs/polars#22248), not 1.5 as previously pinned, so 1.5–1.28 installs crashed with a raw `TypeError` on `round(x, p>0)` under `engine='polars'`; (2) `round(x, p>308)` is the identity on both engines (a float64 has no digits there) instead of pandas raising through an unclear decline while polars returned identity — parity restored, `10.0**p` overflow guarded; (3) polars `round(x, p>0)` normalizes `-0.0` to `+0.0` like the pandas kernel (`round(-0.04, 1)` was `0.0` on pandas vs `-0.0` on polars — invisible to value equality, pinned by a sign-bit test); (4) documented the precision>0 decimal-string deviation vs neo4j (`round(2.675, 2)` = `2.67` binary-double here vs `2.68` BigDecimal there) and added deterministic tie/hazard matrix cases so ties actually reach cuDF/polars-gpu (the fixture's normal-distribution floats never tied). - **GFQL `materialize_nodes()` on an edges-only graph under `engine='polars'`**: crashed with `AttributeError: 'Series' object has no attribute 'drop_duplicates'` — the node-id derivation used pandas-only `.drop_duplicates()` / `.reset_index()` / `.rename(columns=)` on a polars Series/DataFrame. Now polars-aware (`.unique(maintain_order=True)`, `.select().rename({...})`), matching the pandas oracle. Surfaced by any `let()` DAG on an edges-only graph under Polars (the DAG pre-materializes nodes). pandas/cuDF paths unchanged. - **CI: `compute_networkx` HITS test was scipy-version flaky**: `test_compute_networkx_hits_outputs_hubs_and_authorities` used a pure directed 3-cycle (`a→b→c→a`), whose adjacency is a permutation matrix (all singular values equal). networkx HITS computes scores via scipy `svds(k=1)`, which then returns an arbitrary vector from that fully-degenerate singular space; when its components sum to ~0, HITS's `h /= h.sum()` normalization blows up to `inf`/`nan`. Which vector `svds` returns is scipy/LAPACK-version dependent, so the test passed on the pinned scipy but failed on the `nx-upper-scipy` CI profile's newer scipy — a latent flake, not a code regression. Switched to a graph with a well-defined hub/authority structure (`a→b`, `a→c`, `b→c`), whose dominant singular vector is unique (Perron-Frobenius) and therefore version-stable. Test-only change. diff --git a/bin/coverage_audit.py b/bin/coverage_audit.py index 980e8fdd6f..8a5db836d2 100755 --- a/bin/coverage_audit.py +++ b/bin/coverage_audit.py @@ -614,8 +614,23 @@ def main(argv: Optional[Sequence[str]] = None) -> int: print(f"Wrote {json_path}") if exit_code: print(f"pytest failed with exit code {exit_code}", file=sys.stderr) - if any(check.status == "fail" for check in baseline_checks): + failing_baseline_checks = [check for check in baseline_checks if check.status == "fail"] + if failing_baseline_checks: print("per-file coverage baseline failed", file=sys.stderr) + for check in failing_baseline_checks: + actual = "missing" if check.actual_percent is None else f"{check.actual_percent:.2f}%" + delta = "n/a" if check.delta_percent is None else f"{check.delta_percent:+.2f}%" + print( + " {path}: actual={actual} floor={floor:.2f}% tolerance={tolerance:.2f}% delta={delta} reason={reason}".format( + path=check.path, + actual=actual, + floor=check.min_percent, + tolerance=check.tolerance_percent, + delta=delta, + reason=check.reason, + ), + file=sys.stderr, + ) return exit_code or 3 return exit_code diff --git a/bin/test-polars.sh b/bin/test-polars.sh index 6d7965cee6..a6a6f5b936 100755 --- a/bin/test-polars.sh +++ b/bin/test-polars.sh @@ -3,7 +3,7 @@ set -ex # Run from project root # - Extra args are passed through to the pytest phase -# - Set POLARS_COV=1 to collect coverage over the graphistry package; the coverage +# - Set POLARS_COV=1 to collect coverage over graphistry/compute; the coverage # data file location is taken from $COVERAGE_FILE (as the CI py3.12 lane sets it) # - Non-zero exit code on fail @@ -17,25 +17,18 @@ POLARS_TEST_FILES=( graphistry/tests/compute/gfql/test_engine_polars_hop.py graphistry/tests/compute/gfql/test_engine_polars_chain.py graphistry/tests/compute/gfql/test_engine_polars_row_pipeline.py + graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py graphistry/tests/compute/gfql/test_engine_polars_cypher_conformance.py graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py graphistry/tests/compute/gfql/test_conformance_ledger.py # index tests exercise the seeded-index hook in the polars hop entry (hop.py) — without # them the hook dominates the now-thin file and trips its per-file coverage floor graphistry/tests/compute/gfql/index/test_index.py - # Engine coercion tests: the polars paths (df_to_engine, _pl_nan_to_null identity) only - # run where polars is installed, and this lane is the only coverage-collecting lane - # with polars — the core py3.14 lane has no polars extra, so those tests skip there - graphistry/tests/compute/test_engine_coercion.py ) -# The whole graphistry package is measured here (not just graphistry/compute) because -# polars-only branches outside compute/ (e.g. Engine._pl_nan_to_null) are unreachable in -# the core coverage lane (no polars extra); coverage sources must be dirs/packages, and a -# dotted module source (graphistry.Engine) breaks numpy under pytest COV_ARGS=() if [ -n "${POLARS_COV:-}" ]; then - COV_ARGS=(--cov=graphistry --cov-report=) + COV_ARGS=(--cov=graphistry/compute --cov-report=) fi python -B -m pytest -vv "${COV_ARGS[@]}" "${POLARS_TEST_FILES[@]}" "$@" @@ -44,7 +37,7 @@ python -B -m pytest -vv "${COV_ARGS[@]}" "${POLARS_TEST_FILES[@]}" "$@" # appended into the same coverage data file when POLARS_COV=1 (CI audit reads it) COV_APPEND_ARGS=() if [ -n "${POLARS_COV:-}" ]; then - COV_APPEND_ARGS=(--cov=graphistry --cov-report= --cov-append) + COV_APPEND_ARGS=(--cov=graphistry/compute --cov-report= --cov-append) fi python -B -m pytest -vv "${COV_APPEND_ARGS[@]}" \ graphistry/tests/compute/gfql/cypher/test_lowering.py -k polars diff --git a/graphistry/Engine.py b/graphistry/Engine.py index a19c8f1449..a3d93c2939 100644 --- a/graphistry/Engine.py +++ b/graphistry/Engine.py @@ -214,25 +214,11 @@ def _pl_nan_to_null(df): polars / Arrow / cuDF input carrying genuine NaN is treated as MISSING like the pandas oracle (which skipna/dropna's NaN). Without this, ``engine='polars'`` on a frame with a real NaN keeps rows a filter/aggregation should drop (silent divergence from pandas). - No-op when there are no float columns. - - Identity-stable: returns the *same* frame object when no float column actually carries a - NaN. ``fill_nan(None)`` on a NaN-free column is a value-level no-op but still yields a - FRESH frame; that identity churn defeats frame-identity caches keyed on ``source_ref is - df`` (the #1658 CSR index) -- see #1726. So we probe for real NaN presence per float - column (``is_nan`` only applies to float dtypes, hence the schema gate) and rebuild only - when -- and only the columns where -- a NaN is genuinely present. Values are identical to - the old unconditional rewrite (``fill_nan(None)`` never touches non-NaN entries).""" + No-op when there are no float columns.""" import polars as pl float_cols = [c for c, dt in df.schema.items() if dt in (pl.Float32, pl.Float64)] if not float_cols: return df - if isinstance(df, pl.DataFrame): - nan_cols = [c for c in float_cols if df.get_column(c).is_nan().any()] - if not nan_cols: - return df - return df.with_columns([pl.col(c).fill_nan(None) for c in nan_cols]) - # LazyFrame (rare): no cheap eager NaN probe, keep the original unconditional rewrite. return df.with_columns([pl.col(c).fill_nan(None) for c in float_cols]) diff --git a/graphistry/compute/gfql/index/__init__.py b/graphistry/compute/gfql/index/__init__.py index f58588bb79..7ee6cbf4f8 100644 --- a/graphistry/compute/gfql/index/__init__.py +++ b/graphistry/compute/gfql/index/__init__.py @@ -23,9 +23,7 @@ is_index_op, is_index_op_json, ) from .cypher_ddl import parse_index_ddl, looks_like_index_ddl -from .cost import ( - cost_gate_frac, cost_gate_min_frontier, reset_cost_gate_frac, set_cost_gate_frac, -) +from .cost import cost_gate_frac, reset_cost_gate_frac, set_cost_gate_frac from .explain import GfqlExplainReport __all__ = [ @@ -40,7 +38,6 @@ "CreateIndex", "DropIndex", "ShowIndexes", "IndexOp", "apply_index_op", "index_op_from_json", "is_index_op", "is_index_op_json", "parse_index_ddl", "looks_like_index_ddl", - "cost_gate_frac", "cost_gate_min_frontier", "reset_cost_gate_frac", - "set_cost_gate_frac", + "cost_gate_frac", "reset_cost_gate_frac", "set_cost_gate_frac", "GfqlExplainReport", ] diff --git a/graphistry/compute/gfql/index/api.py b/graphistry/compute/gfql/index/api.py index f5296bd911..bc6f751e5a 100644 --- a/graphistry/compute/gfql/index/api.py +++ b/graphistry/compute/gfql/index/api.py @@ -21,7 +21,7 @@ ) from .build import build_adjacency_index, build_node_id_index from .traverse import index_seeded_hop -from .cost import cost_gate_frac, cost_gate_min_frontier, seed_deg_sum, seed_id_array +from .cost import cost_gate_frac, seed_deg_sum, seed_id_array from .policy import IndexPolicy, validate_index_policy from .types import ( AdjacencyIndexKind, EdgeIndexDirection, HopDirection, IndexKind, @@ -424,15 +424,10 @@ def _bail(reason: str) -> Optional[Plottable]: # Cost gate: if the frontier covers a large fraction of distinct sources, the # scan path is competitive — fall back (avoids index overhead on bulk-ish hops). - # Small-frontier floor: a frontier of <= cost_gate_min_frontier() seeds always - # indexes — the frac gate scales with n_keys, so on small/low-cardinality slices - # it would otherwise send even a single-seed hop to the O(E) scan. Pure routing - # heuristic either way: index and scan return identical results. idx0 = cast(Optional[AdjacencyIndex], registry.get_valid( EDGE_OUT_ADJ if direction != "reverse" else EDGE_IN_ADJ, g._edges, (src, dst), engine )) frac = cost_gate_frac(engine) - min_frontier = cost_gate_min_frontier() if trace and idx0 is not None: # Free fanout estimate (Σ seed degree) from the CSR offsets — the planner # signal the report wants EXPLAIN to surface (not just used-index yes/no). @@ -442,19 +437,16 @@ def _bail(reason: str) -> Optional[Plottable]: diag["seed_deg_sum"] = deg_sum diag["est_result_rows"] = deg_sum diag["threshold_frac"] = frac - diag["min_frontier_floor"] = min_frontier if idx0 is None: # required direction not resident (undirected needs both); let driver decide pass elif resolved_policy != "force": try: frontier_n = int(nodes.shape[0]) - if (frontier_n > min_frontier - and idx0.n_keys > 0 and frontier_n >= frac * idx0.n_keys): + if idx0.n_keys > 0 and frontier_n >= frac * idx0.n_keys: return _bail( f"frontier {frontier_n} >= {frac}*n_keys " - f"({frac * idx0.n_keys:.0f}) and > floor ({min_frontier}) " - f"-> scan cheaper" + f"({frac * idx0.n_keys:.0f}) -> scan cheaper" ) except (AttributeError, TypeError, ValueError): pass diff --git a/graphistry/compute/gfql/index/cost.py b/graphistry/compute/gfql/index/cost.py index bd8ca53862..754db74482 100644 --- a/graphistry/compute/gfql/index/cost.py +++ b/graphistry/compute/gfql/index/cost.py @@ -17,46 +17,6 @@ _COST_GATE_FRAC_DEFAULT = 0.02 _COST_GATE_FRAC_OVERRIDES: Dict[Engine, float] = {} -# Absolute small-frontier floor: at or below this many seed rows the planner NEVER -# bails to scan on the frac gate. The frac gate scales with distinct-key cardinality, -# so on small / low-cardinality edge slices (e.g. per-edge-type homogeneous frames: -# n_keys <= 1/0.02 = 50 at the polars/cudf frac) even a single-node seed trips it and -# the hop scans O(E) despite a resident O(degree) index. A frontier of <= K seeds -# bounds CSR lookup work to K searchsorted probes plus the matching-row gather, -# avoiding that fractional-gate artifact. The actual index/scan crossover remains -# engine- and data-dependent: K is a conservative, environment-tunable default, -# not a measured universal threshold. Constant (not a function of n_keys) on -# purpose: scaling with n_keys is the frac gate's job; the floor bounds absolute -# per-hop probe work where frac*n_keys collapses below a handful of seeds. Uniform -# across engines (pandas's 0.5 frac only overlaps the floor when n_keys <= 2*K). -# Purely a routing heuristic: index and scan return identical results. -_COST_GATE_MIN_FRONTIER_DEFAULT = 16 - - -def cost_gate_min_frontier() -> int: - """Return the absolute small-frontier floor for the index-vs-scan cost gate. - - Frontiers of at most this many seeds always take a resident index, regardless - of the frac gate. ``0`` disables the floor (pure frac gating). Env-overridable - via ``GFQL_INDEX_COST_GATE_MIN_FRONTIER`` for benchmark/diagnostic tuning. - """ - raw = os.environ.get("GFQL_INDEX_COST_GATE_MIN_FRONTIER") - if raw is None or raw == "": - return _COST_GATE_MIN_FRONTIER_DEFAULT - try: - val = int(raw) - except ValueError as ex: - raise ValueError( - f"Invalid GFQL_INDEX_COST_GATE_MIN_FRONTIER={raw!r}: " - f"expected a non-negative integer" - ) from ex - if val < 0: - raise ValueError( - f"Invalid GFQL_INDEX_COST_GATE_MIN_FRONTIER={raw!r}: " - f"expected a non-negative integer" - ) - return val - def _validate_cost_gate_frac(frac: float) -> float: if not 0.0 < frac <= 1.0: diff --git a/graphistry/compute/gfql/index/types.py b/graphistry/compute/gfql/index/types.py index 47a088b73c..af3aa800f6 100644 --- a/graphistry/compute/gfql/index/types.py +++ b/graphistry/compute/gfql/index/types.py @@ -30,7 +30,6 @@ class IndexTraceStep(TypedDict, total=False): seed_deg_sum: Optional[int] est_result_rows: Optional[int] threshold_frac: float - min_frontier_floor: int IndexTrace = List[IndexTraceStep] diff --git a/graphistry/compute/gfql/lazy/engine/polars/chain.py b/graphistry/compute/gfql/lazy/engine/polars/chain.py index bfe469b056..72b8769f33 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/chain.py +++ b/graphistry/compute/gfql/lazy/engine/polars/chain.py @@ -361,32 +361,28 @@ def _try_native_row_op(g_cur, op): from .search import search_any_polars fn = getattr(op, "function", None) - if ( - fn == "rows" - and op.params.get("binding_ops") is not None - and op.params.get("alias_endpoints") is None - ): - # Multi-alias bindings table (#1709): native for fixed-length connected - # patterns. A decline must fall through to the pre-existing correlated - # pattern handler below (EXISTS/searchAny); returning None here would turn - # those already-native shapes into an NIE. - bindings_result = binding_rows_polars( - g_cur, op.params["binding_ops"], op.params.get("attach_prop_aliases") - ) - if bindings_result is not None: - return bindings_result + if fn == "rows" and op.params.get("binding_ops") is not None: + # #1731: single-entity boundary rows (MATCH (n) / EXISTS seeds) are handled by + # the pattern-apply helper; try that narrow shape first. + if op.params.get("source") is None: + out = rows_binding_ops_polars(g_cur, op.params["binding_ops"]) + if out is not None: + return out + # #1730 gate: only take the multi-alias bindings table when alias_endpoints is + # absent (the alias-endpoints shape is handled elsewhere and must fall through). + if op.params.get("alias_endpoints") is None: + # Multi-alias bindings table (#1709): native for fixed-length connected + # patterns. A decline must fall through to the pre-existing correlated + # pattern handler below (EXISTS/searchAny); returning None here would turn + # those already-native shapes into an NIE. + bindings_result = binding_rows_polars( + g_cur, op.params["binding_ops"], op.params.get("attach_prop_aliases") + ) + if bindings_result is not None: + return bindings_result if _call_native_on_polars(op): # frame ops (rows/limit/skip/distinct/drop_cols) — engine-polymorphic return op.execute(g=g_cur, prev_node_wavefront=None, target_wave_front=None, engine=Engine.POLARS) - # correlated pattern-existence family (EXISTS { } / pattern predicates): native - # via chain_polars-computed key sets; unsupported shapes return None -> honest NIE. - if ( - fn == "rows" - and op.params.get("binding_ops") is not None - and op.params.get("alias_endpoints") is None - and op.params.get("source") is None - ): - return rows_binding_ops_polars(g_cur, op.params["binding_ops"]) if fn == "semi_apply_mark": # required params are safelist-validated — direct indexing (an or-default # here could only mask an unvalidated call); neq is the optional one. diff --git a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py index c7c33a940d..17b071446b 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py +++ b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py @@ -968,6 +968,23 @@ def _names(lf: pl.LazyFrame) -> List[str]: oriented = edges_f.rename({join_col: "__from__", result_col: "__to__"}) if payload_renames: oriented = oriented.rename(payload_renames) + + next_op = ops[edge_idx + 1] + if not isinstance(next_op, ASTNode): + return None + next_nodes = filter_by_dict_polars(nodes_lf, next_op.filter_dict) + next_node_ids = next_nodes.select(node_id).unique() + if not sem.is_multihop: + # Filter endpoint candidates before joining from the current state. + # For graph-bench q5/q6/q7, pushed Interest/City predicates make + # this turn an all-edges scan into a small-domain edge semi-join. + oriented = oriented.join( + next_node_ids, + left_on="__to__", + right_on=node_id, + how="semi", + ) + # Column collision between edge payload and accumulated state → decline # (pandas resolves via merge suffixes; unreferenced-by-queries either way). overlap = (set(_names(oriented)) - {"__from__"}) & set(_names(state)) @@ -1012,12 +1029,8 @@ def _names(lf: pl.LazyFrame) -> List[str]: .rename({"__to__": "__current__"}) ) - next_op = ops[edge_idx + 1] - if not isinstance(next_op, ASTNode): - return None - next_nodes = filter_by_dict_polars(nodes_lf, next_op.filter_dict) state = state.join( - next_nodes.select(node_id).unique(), + next_node_ids, left_on="__current__", right_on=node_id, how="semi", @@ -1037,7 +1050,10 @@ def _names(lf: pl.LazyFrame) -> List[str]: continue # properties unreferenced — keep only the bare id column lookup_src = alias_frames[alias] lookup = lookup_src.select( - [pl.col(node_id), pl.col(node_id).alias(f"{alias}.{node_id}")] + [ + pl.col(node_id), + pl.col(node_id).alias(f"{alias}.{node_id}"), + ] + [ pl.col(col).alias(f"{alias}.{col}") for col in _names(lookup_src) diff --git a/graphistry/compute/gfql/row/frame_ops.py b/graphistry/compute/gfql/row/frame_ops.py index deed488a97..e734793096 100644 --- a/graphistry/compute/gfql/row/frame_ops.py +++ b/graphistry/compute/gfql/row/frame_ops.py @@ -27,6 +27,7 @@ def _gfql_binding_ops_row_table( self, binding_ops: List[Dict[str, JSONVal]], alias_prefilters: Optional[AliasPrefilters] = None, + attach_prop_aliases: Optional[List[str]] = None, ) -> "Plottable": ... def _gfql_bindings_row_table(self, alias_endpoints: Any) -> "Plottable": ... @@ -148,7 +149,7 @@ def coerce_non_negative_int(value: Any, op_name: str) -> int: def rows( - ctx: Any, + ctx: RowPipelineCtx, table: Optional[str] = None, source: Optional[str] = None, alias_endpoints: Optional[Dict[str, str]] = None, diff --git a/graphistry/compute/gfql/row/prefilter.py b/graphistry/compute/gfql/row/prefilter.py index b92a3ee0ae..9c859e1750 100644 --- a/graphistry/compute/gfql/row/prefilter.py +++ b/graphistry/compute/gfql/row/prefilter.py @@ -27,24 +27,10 @@ def _is_alias_prefilter_spec(value: object) -> bool: return False kind = value.get("kind") if kind == "expr": - return ( - set(value).issubset({"kind", "text"}) - and isinstance(value.get("text"), str) - ) + return isinstance(value.get("text"), str) if kind == "search_any": - if not set(value).issubset( - {"kind", "term", "case_sensitive", "regex", "columns"} - ): - return False - if not isinstance(value.get("term"), str): - return False - if "case_sensitive" in value and not isinstance(value["case_sensitive"], bool): - return False - if "regex" in value and not isinstance(value["regex"], bool): - return False - if "columns" in value and not _is_string_list(value["columns"]): - return False - return True + columns = value.get("columns") + return isinstance(value.get("term"), str) and (columns is None or _is_string_list(columns)) return False diff --git a/graphistry/compute/gfql_fast_paths.py b/graphistry/compute/gfql_fast_paths.py new file mode 100644 index 0000000000..3dc8405693 --- /dev/null +++ b/graphistry/compute/gfql_fast_paths.py @@ -0,0 +1,1723 @@ +"""Fast-path specialization helpers for the unified GFQL executor. + +Extracted verbatim from gfql_unified.py (#1731) to keep that orchestrator readable: the +connected-join single/two-star fast paths and the grouped-aggregate / two-hop-count fast +paths. Pure code move, no behavior change. gfql_unified imports from here (one direction); +this module imports only leaf modules (no back-edge into gfql_unified). +""" + +"""GFQL unified entrypoint for chains, DAGs, and local string-compiled queries.""" +# ruff: noqa: E501 + +from dataclasses import replace +import pandas as pd +from types import MappingProxyType +from typing import Any, Dict, List, Literal, Mapping, Optional, Sequence, Set, Tuple, Union, cast +from graphistry.Plottable import Plottable +from graphistry.Engine import Engine, EngineAbstract, POLARS_ENGINES, df_concat, df_cons, df_to_engine, df_unique, resolve_engine +from graphistry.util import setup_logger +from .ast import ASTObject, ASTLet, ASTNode, ASTEdge, ASTCall +from .chain import Chain, chain as chain_impl +from .gfql.query_types import GFQLQuery +from .chain_let import chain_let as chain_let_impl +from .execution_context import ExecutionContext +from .gfql.policy import ( + CompileSummary, + PolicyContext, + PolicyException, + PolicyFunction, + PolicyDict, + QueryType, + expand_policy +) +from graphistry.compute.gfql.same_path_types import ( + NODE_IDENTITY_COLUMN, + WhereComparison, + normalize_where_entries, + parse_where_json, +) +from graphistry.compute.exceptions import ErrorCode, GFQLValidationError +from graphistry.compute.gfql.cypher.parser import parse_cypher +from graphistry.compute.gfql.cypher.lowering import ( + ConnectedMatchJoinPlan, + CompiledCypherGraphQuery, + CompiledCypherQuery, + CompiledCypherUnionQuery, + CompiledGraphResidualFilter, + ConnectedOptionalMatchPlan, + compile_cypher_query, +) +from graphistry.compute.filter_by_dict import _node_dtypes_for_pushdown +from graphistry.compute.gfql.cypher.reentry.execution import ( + REENTRY_DUPLICATE_CARRIED_ROWS_REASON as _REENTRY_DUPLICATE_CARRIED_ROWS_REASON, + REENTRY_WHOLE_ROW_SUGGESTION as _REENTRY_WHOLE_ROW_SUGGESTION, + apply_optional_reentry_null_fill as _apply_optional_reentry_null_fill, + compiled_query_freeform_reentry_state as _compiled_query_freeform_reentry_state, + compiled_query_reentry_state as _compiled_query_reentry_state, + compiled_query_scalar_reentry_state as _compiled_query_scalar_reentry_state, + freeform_broadcast_row_to_nodes as _freeform_broadcast_row_to_nodes, + reentry_validation_error as _reentry_validation_error, + union_scalar_reentry_results as _union_scalar_reentry_results, +) +from graphistry.compute.gfql.cypher.call_procedures import execute_cypher_call +from graphistry.compute.gfql.cypher.result_postprocess import ( + apply_result_projection, + entity_projection_meta_entry as _entity_projection_meta_entry, +) +from graphistry.compute.gfql.df_executor import ( + DFSamePathExecutor, + build_same_path_inputs, + execute_same_path_chain, +) +from graphistry.compute.dataframe import ( + binding_join_columns as _binding_join_columns, + connected_inner_join_rows as _connected_inner_join_rows, + joined_alias_columns as _joined_alias_columns, + joined_hidden_scalar_columns as _joined_hidden_scalar_columns, +) +from graphistry.compute.filter_by_dict import filter_by_dict +from graphistry.compute.gfql.ir.compilation import PhysicalPlan, PlanContext +from graphistry.compute.gfql.ir.logical_plan import LogicalPlan +from graphistry.compute.gfql.physical_planner import PhysicalPlanner +from graphistry.compute.gfql.passes import DEFAULT_LOGICAL_PASSES, DEFAULT_TIER2_PASSES, PassManager +from graphistry.compute.gfql.row.pipeline import _RowPipelineAdapter, is_row_pipeline_call +from graphistry.compute.gfql.search_any import search_any_mask +from graphistry.compute.typing import DataFrameT, SeriesT, NodeDtypes +from graphistry.compute.util.generate_safe_column_name import generate_safe_column_name +from graphistry.compute.validate.validate_schema import validate_chain_schema +from graphistry.compute.gfql_validate import gfql_validate as gfql_preflight_validate +from graphistry.otel import otel_traced, otel_detail_enabled + + +def _is_connected_fast_single_hop(edge_op: ASTEdge) -> bool: + return ( + getattr(edge_op, "direction", None) == "forward" + and getattr(edge_op, "hops", None) in (None, 1) + and getattr(edge_op, "min_hops", None) is None + and getattr(edge_op, "max_hops", None) is None + and getattr(edge_op, "output_min_hops", None) is None + and getattr(edge_op, "output_max_hops", None) is None + and getattr(edge_op, "label_node_hops", None) is None + and getattr(edge_op, "label_edge_hops", None) is None + and not bool(getattr(edge_op, "label_seeds", False)) + and not bool(getattr(edge_op, "to_fixed_point", False)) + and getattr(edge_op, "source_node_match", None) is None + and getattr(edge_op, "destination_node_match", None) is None + and getattr(edge_op, "source_node_query", None) is None + and getattr(edge_op, "destination_node_query", None) is None + and getattr(edge_op, "edge_query", None) is None + and getattr(edge_op, "_name", None) is None + and not bool(getattr(edge_op, "include_zero_hop_seed", False)) + ) + + + + +_CACHE_MISSING = object() + + +def _connected_join_filter_value_cache_key(value: Any) -> Optional[Tuple[str, str]]: + if isinstance(value, bool): + return "bool", "true" if value else "false" + if isinstance(value, str): + return "str", value + if isinstance(value, int) and not isinstance(value, bool): + return "int", str(value) + if isinstance(value, float): + return "float", repr(value) + if value is None: + return "none", "" + + predicate_value = getattr(value, "val", _CACHE_MISSING) + if predicate_value is not _CACHE_MISSING: + scalar_key = _connected_join_filter_value_cache_key(predicate_value) + if scalar_key is None: + return None + return f"{type(value).__module__}.{type(value).__qualname__}", f"{scalar_key[0]}:{scalar_key[1]}" + + regex_pattern = getattr(value, "pat", _CACHE_MISSING) + regex_case = getattr(value, "case", _CACHE_MISSING) + regex_flags = getattr(value, "flags", _CACHE_MISSING) + regex_na = getattr(value, "na", _CACHE_MISSING) + if ( + regex_pattern is not _CACHE_MISSING + and regex_case is not _CACHE_MISSING + and regex_flags is not _CACHE_MISSING + and regex_na is not _CACHE_MISSING + ): + pattern_key = _connected_join_filter_value_cache_key(regex_pattern) + case_key = _connected_join_filter_value_cache_key(regex_case) + flags_key = _connected_join_filter_value_cache_key(regex_flags) + na_key = _connected_join_filter_value_cache_key(regex_na) + if pattern_key is None or case_key is None or flags_key is None or na_key is None: + return None + return ( + f"{type(value).__module__}.{type(value).__qualname__}", + f"pat={pattern_key[0]}:{pattern_key[1]}|case={case_key[0]}:{case_key[1]}|flags={flags_key[0]}:{flags_key[1]}|na={na_key[0]}:{na_key[1]}", + ) + + predicates = getattr(value, "predicates", _CACHE_MISSING) + if predicates is not _CACHE_MISSING and isinstance(predicates, (list, tuple)): + child_keys = [] + for predicate in predicates: + child_key = _connected_join_filter_value_cache_key(predicate) + if child_key is None: + return None + child_keys.append(f"{child_key[0]}:{child_key[1]}") + return f"{type(value).__module__}.{type(value).__qualname__}", "|".join(child_keys) + + return None + + +def _connected_join_simple_filter_cache_key(filter_dict: Optional[dict]) -> Optional[Tuple[Tuple[str, str, str], ...]]: + if not filter_dict: + return () + items: List[Tuple[str, str, str]] = [] + for key, value in filter_dict.items(): + if not isinstance(key, str): + return None + value_key = _connected_join_filter_value_cache_key(value) + if value_key is None: + return None + items.append((key, value_key[0], value_key[1])) + return tuple(sorted(items)) + + +def _connected_join_cached_node_filter( + base_graph: Plottable, + nodes_obj: DataFrameT, + node_match: Optional[dict], + *, + engine: Engine, + cache_store: Optional[Dict[str, Any]] = None, +) -> DataFrameT: + cache_key = _connected_join_simple_filter_cache_key(node_match) + if cache_key is None: + nodes = df_to_engine(nodes_obj, engine) + if engine in POLARS_ENGINES: + from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars + return cast(DataFrameT, filter_by_dict_polars(nodes, node_match)) + return filter_by_dict(nodes, node_match, engine=EngineAbstract(engine.value)) + + cache_attr = "_gfql_connected_join_node_filter_cache" + # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's + # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale + # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. + cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None + full_key = (id(nodes_obj), engine.value, cache_key) + if cache is not None and full_key in cache: + return cast(DataFrameT, cache[full_key]) + + nodes = df_to_engine(nodes_obj, engine) + if engine in POLARS_ENGINES: + from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars + filtered = cast(DataFrameT, filter_by_dict_polars(nodes, node_match)) + else: + filtered = filter_by_dict(nodes, node_match, engine=EngineAbstract(engine.value)) + if cache is not None: + cache[full_key] = filtered + return cast(DataFrameT, filtered) + + +def _connected_join_cached_node_ids( # pragma: no cover - polars-only, covered by polars lane + base_graph: Plottable, + nodes_obj: DataFrameT, + node_matches: Sequence[Optional[dict]], + *, + node_col: str, + engine: Engine, + cache_store: Optional[Dict[str, Any]] = None, +) -> Optional[DataFrameT]: + if engine not in POLARS_ENGINES: + return None + + match_keys: List[Tuple[Tuple[str, str, str], ...]] = [] + for node_match in node_matches: + match_key = _connected_join_simple_filter_cache_key(node_match) + if match_key is None: + return None + match_keys.append(match_key) + + cache_attr = "_gfql_connected_join_node_ids_cache" + # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's + # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale + # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. + cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None + full_key = (id(nodes_obj), engine.value, node_col, tuple(match_keys)) + if cache is not None and full_key in cache: + return cast(DataFrameT, cache[full_key]) + + from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars + + nodes = df_to_engine(nodes_obj, engine) + filtered = nodes + for node_match in node_matches: + filtered = cast(DataFrameT, filter_by_dict_polars(filtered, node_match)) + ids = cast(DataFrameT, filtered.select(node_col).unique()) + if cache is not None: + cache[full_key] = ids + return ids + + +def _connected_join_cached_edge_filter( + base_graph: Plottable, + edges_obj: DataFrameT, + edge_match: Optional[dict], + *, + engine: Engine, + cache_store: Optional[Dict[str, Any]] = None, +) -> DataFrameT: + cache_key = _connected_join_simple_filter_cache_key(edge_match) + if cache_key is None: + edges = df_to_engine(edges_obj, engine) + if engine in POLARS_ENGINES: + from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars + return cast(DataFrameT, filter_by_dict_polars(edges, edge_match)) + return filter_by_dict(edges, edge_match, engine=EngineAbstract(engine.value)) + + cache_attr = "_gfql_connected_join_edge_filter_cache" + # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's + # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale + # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. + cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None + full_key = (id(edges_obj), engine.value, cache_key) + if cache is not None and full_key in cache: + return cast(DataFrameT, cache[full_key]) + + edges = df_to_engine(edges_obj, engine) + if engine in POLARS_ENGINES: + from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars + filtered = cast(DataFrameT, filter_by_dict_polars(edges, edge_match)) + else: + filtered = filter_by_dict(edges, edge_match, engine=EngineAbstract(engine.value)) + if cache is not None: + cache[full_key] = filtered + return cast(DataFrameT, filtered) + + +def _connected_join_cached_singleton_dst_source_counts( # pragma: no cover + base_graph: Plottable, + edges_obj: DataFrameT, + edge_domain: DataFrameT, + edge_match: Optional[dict], + singleton_dst: Any, + *, + src_col: str, + dst_col: str, + engine: Engine, + cache_store: Optional[Dict[str, Any]] = None, +) -> Optional[DataFrameT]: + edge_key = _connected_join_simple_filter_cache_key(edge_match) + dst_key = _connected_join_filter_value_cache_key(singleton_dst) + if edge_key is None or dst_key is None: + return None + + cache_attr = "_gfql_connected_join_singleton_dst_source_counts_cache" + # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's + # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale + # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. + cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None + full_key = (id(edges_obj), engine.value, src_col, dst_col, edge_key, dst_key) + if cache is not None and full_key in cache: + return cast(DataFrameT, cache[full_key]) + + if engine in POLARS_ENGINES: # pragma: no cover - polars-only, covered by polars lane + import polars as pl + counts = cast( + DataFrameT, + edge_domain + .filter(pl.col(dst_col) == singleton_dst) + .group_by(src_col) + .len("__left_count__"), + ) + else: + filtered = edge_domain[edge_domain[dst_col] == singleton_dst] + counts = cast(DataFrameT, filtered.groupby(src_col, sort=False).size().reset_index(name="__left_count__")) + + if cache is not None: + cache[full_key] = counts + return counts + + +def _connected_join_cached_first_arm_shared_counts( # pragma: no cover - polars-only, covered by polars lane + base_graph: Plottable, + nodes_obj: DataFrameT, + edges_obj: DataFrameT, + first_edges: DataFrameT, + shared_ids: DataFrameT, + first_edge_match: Optional[dict], + first_start_match: Optional[dict], + second_start_match: Optional[dict], + singleton_dst: Any, + *, + shared_alias: str, + src_col: str, + dst_col: str, + engine: Engine, + cache_store: Optional[Dict[str, Any]] = None, +) -> Optional[DataFrameT]: + if engine not in POLARS_ENGINES: + return None + + edge_key = _connected_join_simple_filter_cache_key(first_edge_match) + first_start_key = _connected_join_simple_filter_cache_key(first_start_match) + second_start_key = _connected_join_simple_filter_cache_key(second_start_match) + dst_key = _connected_join_filter_value_cache_key(singleton_dst) + if edge_key is None or first_start_key is None or second_start_key is None or dst_key is None: + return None + + cache_attr = "_gfql_connected_join_first_arm_shared_counts_cache" + # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's + # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale + # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. + cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None + full_key = ( + id(nodes_obj), + id(edges_obj), + engine.value, + shared_alias, + src_col, + dst_col, + edge_key, + first_start_key, + second_start_key, + dst_key, + ) + if cache is not None and full_key in cache: + return cast(DataFrameT, cache[full_key]) + + import polars as pl + + counts = ( + first_edges + .filter(pl.col(dst_col) == singleton_dst) + .join(shared_ids, left_on=src_col, right_on=shared_alias, how="semi") + .group_by(src_col) + .len("__left_count__") + .rename({src_col: shared_alias}) + ) + if cache is not None: + cache[full_key] = counts + return cast(DataFrameT, counts) + + +def _connected_join_cached_second_arm_group_rows( # pragma: no cover - polars-only, covered by polars lane + base_graph: Plottable, + nodes_obj: DataFrameT, + edges_obj: DataFrameT, + second_edges: DataFrameT, + shared_ids: DataFrameT, + second_leaf_ids: DataFrameT, + second_leaf_nodes: DataFrameT, + first_start_match: Optional[dict], + second_start_match: Optional[dict], + second_leaf_match: Optional[dict], + second_edge_match: Optional[dict], + second_leaf_singleton: Any, + group_prop_refs: List[Tuple[str, str]], + output_group_keys: List[str], + *, + node_col: str, + src_col: str, + dst_col: str, + engine: Engine, + cache_store: Optional[Dict[str, Any]] = None, +) -> Optional[DataFrameT]: + if engine not in POLARS_ENGINES: + return None + + first_start_key = _connected_join_simple_filter_cache_key(first_start_match) + second_start_key = _connected_join_simple_filter_cache_key(second_start_match) + second_leaf_key = _connected_join_simple_filter_cache_key(second_leaf_match) + second_edge_key = _connected_join_simple_filter_cache_key(second_edge_match) + if first_start_key is None or second_start_key is None or second_leaf_key is None or second_edge_key is None: + return None + + prop_key = tuple((str(out_col), str(prop)) for out_col, prop in group_prop_refs) + output_key = tuple(str(key) for key in output_group_keys) + cache_attr = "_gfql_connected_join_second_arm_group_rows_cache" + # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's + # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale + # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. + cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None + full_key = ( + id(nodes_obj), + id(edges_obj), + engine.value, + node_col, + src_col, + dst_col, + first_start_key, + second_start_key, + second_leaf_key, + second_edge_key, + prop_key, + output_key, + ) + if cache is not None and full_key in cache: + return cast(DataFrameT, cache[full_key]) + + import polars as pl + + if second_leaf_singleton is _CACHE_MISSING: + right_base = ( + second_edges + .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") + .join(second_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") + ) + else: + right_base = ( + second_edges + .filter(pl.col(dst_col) == second_leaf_singleton) + .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") + ) + if group_prop_refs: + lookup_key = "__gfql_fast_second_leaf_id__" + lookup_exprs = [pl.col(node_col).alias(lookup_key)] + [pl.col(prop).alias(out_col) for out_col, prop in group_prop_refs] + # Dedup by node identity (matches pandas drop_duplicates(subset=[node_col])); a + # duplicate node row would otherwise multiply the join and over-count. + second_lookup = second_leaf_nodes.select(lookup_exprs).unique(subset=[lookup_key]) + right_base = right_base.join(second_lookup, left_on=dst_col, right_on=lookup_key, how="inner") + right_rows = right_base.select([pl.col(src_col)] + [pl.col(key) for key in output_group_keys]) + if cache is not None: + cache[full_key] = right_rows + return cast(DataFrameT, right_rows) + + +def _two_hop_cached_equal_domain_degree_counts( + base_graph: Plottable, + nodes_obj: DataFrameT, + edges_obj: DataFrameT, + domain_nodes: DataFrameT, + edge_domain: DataFrameT, + *, + node_match: Optional[dict], + edge_match: Optional[dict], + node_col: str, + src_col: str, + dst_col: str, + engine: Engine, +) -> Optional[Tuple[DataFrameT, DataFrameT]]: + node_key = _connected_join_simple_filter_cache_key(node_match) + edge_key = _connected_join_simple_filter_cache_key(edge_match) + if node_key is None or edge_key is None: + return None + + cache_attr = "_gfql_two_hop_equal_domain_degree_counts_cache" + cache = getattr(base_graph, cache_attr, None) + if not isinstance(cache, dict): + cache = {} + try: + setattr(base_graph, cache_attr, cache) + except Exception: + cache = None + full_key = (id(nodes_obj), id(edges_obj), engine.value, node_col, src_col, dst_col, node_key, edge_key) + if cache is not None and full_key in cache: + return cast(Tuple[DataFrameT, DataFrameT], cache[full_key]) + + if engine in POLARS_ENGINES: + domain_ids = domain_nodes.select(node_col).unique() + filtered_edges = ( + edge_domain + .join(domain_ids, left_on=src_col, right_on=node_col, how="semi") + .join(domain_ids, left_on=dst_col, right_on=node_col, how="semi") + ) + counts = ( + cast(DataFrameT, filtered_edges.group_by(dst_col).len("__in_count__")), + cast(DataFrameT, filtered_edges.group_by(src_col).len("__out_count__")), + ) + else: + domain_ids = domain_nodes[node_col].drop_duplicates() + filtered_edges = edge_domain[edge_domain[src_col].isin(domain_ids) & edge_domain[dst_col].isin(domain_ids)] + counts = ( + cast(DataFrameT, filtered_edges.groupby(dst_col, sort=False).size().reset_index(name="__in_count__")), + cast(DataFrameT, filtered_edges.groupby(src_col, sort=False).size().reset_index(name="__out_count__")), + ) + + if cache is not None: + cache[full_key] = counts + return counts + + +def _connected_join_post_property_columns(plan: ConnectedMatchJoinPlan, alias: str) -> List[str]: + prefix = f"{alias}." + out: List[str] = [] + + def walk(value: Any) -> None: + if isinstance(value, str): + if value.startswith(prefix): + prop = value[len(prefix):] + if prop and prop not in out: + out.append(prop) + return + if isinstance(value, Mapping): + for item in value.values(): + walk(item) + return + if isinstance(value, (list, tuple)): + for item in value: + walk(item) + + for op in plan.post_join_chain.chain: + if isinstance(op, ASTCall): + walk(op.params) + return out + + +def _connected_join_two_star_split_residuals( + plan: ConnectedMatchJoinPlan, + alias_targets: Mapping[str, ASTObject], + materialized_aliases: Set[str], +) -> Optional[Tuple[Dict[str, List[str]], "Chain"]]: + """Split leading post-join ``where_rows`` residuals by single node alias. + + #1729's connected-join lowering emits row predicates it cannot push into ``filter_dict`` + (e.g. ``toLower(i.interest) = toLower('fine dining')``) as leading ``where_rows`` ops in + ``post_join_chain``. The structural fast paths cannot apply a residual to the aggregated + frame, but a residual that references exactly ONE node alias the fast path materializes can + be applied to that alias's node set before counting. Returns ``(alias -> [expr, ...], + remaining_chain)`` when every leading residual is so attributable, or ``None`` when any + leading residual spans >1 alias, references a non-materialized alias, or is not a bare + ``expr`` residual -- the caller then declines to the slow path. + """ + from graphistry.compute.gfql.cypher.lowering import _expr_match_aliases + + ops = list(plan.post_join_chain.chain) + residuals: Dict[str, List[str]] = {} + consumed = 0 + for op in ops: + if not (isinstance(op, ASTCall) and op.function == "where_rows"): + break + params = op.params or {} + expr = params.get("expr") + if params.get("filter_dict") or not isinstance(expr, str): + return None + try: + aliases = _expr_match_aliases( + expr, alias_targets=alias_targets, params=None, field="where", line=0, column=0 + ) + except Exception: + return None + attributable = {alias for alias in aliases if alias in materialized_aliases} + if len(aliases) != 1 or len(attributable) != 1: + return None + residuals.setdefault(next(iter(attributable)), []).append(expr) + consumed += 1 + rest = Chain(ops[consumed:], where=plan.post_join_chain.where) + return residuals, rest + + +def _connected_join_apply_node_residuals( + base_graph: Plottable, + node_frame: DataFrameT, + alias: str, + exprs: Sequence[str], + node_col: str, + *, + engine: Engine, +) -> DataFrameT: + """Filter a fast-path node frame by single-alias post-join residual expressions. + + Reuses the row pipeline's ``where_rows`` evaluator (identical semantics to the slow path, + so toLower/etc. behave exactly as they would post-join) by aliasing the node columns to + ``alias.col`` and dispatching a where_rows chain, then renaming back. ``validate_schema`` is + disabled because the residual references flat ``alias.col`` columns rather than a bound + graph element. + """ + is_polars = "polars" in type(node_frame).__module__ + if is_polars: + aliased = node_frame.rename({col: f"{alias}.{col}" for col in node_frame.columns}) + else: + aliased = node_frame.rename(columns={col: f"{alias}.{col}" for col in node_frame.columns}) + from graphistry.compute.chain import chain as _chain_fn + + aliased_graph = base_graph.nodes(aliased, f"{alias}.{node_col}") + filtered_graph = _chain_fn( + aliased_graph, + [ASTCall("where_rows", {"expr": expr}) for expr in exprs], + engine=EngineAbstract(engine.value), + validate_schema=False, + ) + filtered = cast(DataFrameT, filtered_graph._nodes) + if is_polars: + return cast(DataFrameT, filtered.rename({f"{alias}.{col}": col for col in node_frame.columns})) + return cast(DataFrameT, filtered.rename(columns={f"{alias}.{col}": col for col in node_frame.columns})) + + +def _connected_join_filter_node_frames_by_residuals( + base_graph: Plottable, + residual_map: Mapping[str, Sequence[str]], + frames: Mapping[str, DataFrameT], + node_col: str, + *, + engine: Engine, +) -> Dict[str, DataFrameT]: + """Apply each alias's post-join residuals to its materialized node frame.""" + out: Dict[str, DataFrameT] = dict(frames) + for alias, exprs in residual_map.items(): + if alias in out and exprs: + out[alias] = _connected_join_apply_node_residuals( + base_graph, out[alias], alias, exprs, node_col, engine=engine + ) + return out + + +def _connected_join_two_star_fast_grouped_count( + base_graph: Plottable, + plan: ConnectedMatchJoinPlan, + *, + engine: Engine, + cache_store: Optional[Dict[str, Any]] = None, +) -> Optional[DataFrameT]: + # Per-execution cache scope: a fresh store when none is threaded in, so intra-query reuse + # is preserved without ever persisting caches on the caller's Plottable (BLOCKER 1). + if cache_store is None: + cache_store = {} + if len(plan.pattern_chains) != 2 or len(plan.pattern_shared_node_aliases) != 1: + return None + if len(plan.pattern_shared_node_aliases[0]) != 1: + return None + shared_alias = plan.pattern_shared_node_aliases[0][0] + + parsed = [] + for pattern_chain in plan.pattern_chains: + if pattern_chain.where: + return None + ops = list(pattern_chain.chain) + if len(ops) != 3: + return None + start_op, edge_op, end_op = ops + if not isinstance(start_op, ASTNode) or not isinstance(edge_op, ASTEdge) or not isinstance(end_op, ASTNode): + return None + if getattr(start_op, "query", None) is not None or getattr(end_op, "query", None) is not None: + return None + if not _is_connected_fast_single_hop(edge_op): + return None + start_alias = getattr(start_op, "_name", None) + end_alias = getattr(end_op, "_name", None) + if start_alias != shared_alias or not isinstance(end_alias, str): + return None + parsed.append((start_op, edge_op, end_op, end_alias)) + + first_start, first_edge, first_end, first_end_alias = parsed[0] + second_start, second_edge, second_end, second_end_alias = parsed[1] + if first_end_alias == second_end_alias: + return None + + alias_targets: Dict[str, ASTObject] = { + shared_alias: first_start, + first_end_alias: first_end, + second_end_alias: second_end, + } + materialized_aliases = {shared_alias, first_end_alias, second_end_alias} + split = _connected_join_two_star_split_residuals(plan, alias_targets, materialized_aliases) + if split is None: + return None + residual_map, rest_chain = split + + post_ops = [op for op in rest_chain.chain if isinstance(op, ASTCall)] + if len(post_ops) not in (2, 3, 4): + return None + if post_ops[0].function != "with_" or post_ops[1].function != "group_by": + return None + suffix = post_ops[2:] + order_call: Optional[ASTCall] = None + limit_call: Optional[ASTCall] = None + select_call: Optional[ASTCall] = None + for op in suffix: + if op.function == "order_by" and order_call is None and limit_call is None and select_call is None: + order_call = op + elif op.function == "limit" and limit_call is None and select_call is None: + limit_call = op + elif op.function == "select" and select_call is None: + select_call = op + else: + return None + + with_items_raw = post_ops[0].params.get("items") + group_keys_raw = post_ops[1].params.get("keys") + aggs_raw = post_ops[1].params.get("aggregations") + if not isinstance(with_items_raw, list) or not isinstance(group_keys_raw, list) or not isinstance(aggs_raw, list): + return None + if len(aggs_raw) != 1: + return None + agg = aggs_raw[0] + if not isinstance(agg, (tuple, list)) or len(agg) != 3 or str(agg[1]).lower() != "count": + return None + agg_alias = str(agg[0]) + agg_input = agg[2] + + with_items: Dict[str, Any] = {} + for item in with_items_raw: + if not isinstance(item, (tuple, list)) or len(item) != 2 or not isinstance(item[0], str): + return None + with_items[item[0]] = item[1] + # count(p) over the shared alias now lowers `p` to its identity column + # `p.__gfql_node_id__` (see _connected_join_alias_identity_expr); accept either the + # bare shared alias or that identity form -- the fast count is shared-node multiplicity + # regardless, so the identity rewrite does not change the computation. + shared_identity = f"{shared_alias}.{NODE_IDENTITY_COLUMN}" + if not isinstance(agg_input, str) or with_items.get(agg_input) not in (shared_alias, shared_identity): + return None + + group_keys = [str(key) for key in group_keys_raw] + group_prop_refs: List[Tuple[str, str]] = [] + output_group_keys: List[str] = [] + for key in group_keys: + expr = with_items.get(key) + if key == "__cypher_group__" and expr == 1: + continue + prop_ref = _property_ref(expr, (second_end_alias,)) + if prop_ref is None: + return None + output_group_keys.append(key) + group_prop_refs.append((key, prop_ref[1])) + + order_keys: List[Tuple[str, bool]] = [] + if order_call is not None: + raw_order = order_call.params.get("keys") + if not isinstance(raw_order, list): + return None + available = set(output_group_keys) | {agg_alias} + for item in raw_order: + if not isinstance(item, (tuple, list)) or len(item) != 2 or not isinstance(item[0], str): + return None + if item[0] not in available: + return None + direction = str(item[1]).lower() + if direction not in {"asc", "ascending", "desc", "descending"}: + return None + order_keys.append((item[0], direction in {"desc", "descending"})) + + limit_value: Optional[int] = None + if limit_call is not None: + raw_limit = limit_call.params.get("value") + if not isinstance(raw_limit, int) or raw_limit < 0: + return None + limit_value = raw_limit + + select_items: Optional[List[Tuple[str, str]]] = None + if select_call is not None: + raw_select = select_call.params.get("items") + if not isinstance(raw_select, list): + return None + select_items = [] + available = set(output_group_keys) | {agg_alias} + for item in raw_select: + if not isinstance(item, (tuple, list)) or len(item) != 2 or not isinstance(item[0], str) or not isinstance(item[1], str): + return None + if item[0] not in available: + return None + select_items.append((item[0], item[1])) + + nodes_obj = getattr(base_graph, "_nodes", None) + edges_obj = getattr(base_graph, "_edges", None) + node_col = getattr(base_graph, "_node", None) + src_col = getattr(base_graph, "_source", None) + dst_col = getattr(base_graph, "_destination", None) + if nodes_obj is None or edges_obj is None or node_col is None or src_col is None or dst_col is None: + return None + node_col = str(node_col) + src_col = str(src_col) + dst_col = str(dst_col) + if node_col not in nodes_obj.columns or src_col not in edges_obj.columns or dst_col not in edges_obj.columns: + return None + + nodes_source = cast(DataFrameT, nodes_obj) + nodes = df_to_engine(nodes_source, engine) + edges = cast(DataFrameT, edges_obj) + + if engine in POLARS_ENGINES: # pragma: no cover - polars-only, covered by polars lane + import polars as pl + from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars + + # Residual node filters (e.g. toLower(...)) must be applied to the materialized node + # frames, so residual queries use the direct (non-cached) path -- the id/count caches + # re-derive node sets from filter_dict alone and would drop the residual. + if isinstance(nodes_obj, (pl.DataFrame, pl.LazyFrame)) and not residual_map: + shared_ids = _connected_join_cached_node_ids( + base_graph, + nodes_source, + [cast(Optional[dict], first_start.filter_dict), cast(Optional[dict], second_start.filter_dict)], + node_col=node_col, + engine=engine, + cache_store=cache_store, + ) + first_leaf_ids = _connected_join_cached_node_ids( + base_graph, + nodes_source, + [cast(Optional[dict], first_end.filter_dict)], + node_col=node_col, + engine=engine, + cache_store=cache_store, + ) + second_leaf_ids = _connected_join_cached_node_ids( + base_graph, + nodes_source, + [cast(Optional[dict], second_end.filter_dict)], + node_col=node_col, + engine=engine, + cache_store=cache_store, + ) + first_leaf_nodes = _connected_join_cached_node_filter(base_graph, nodes_source, cast(Optional[dict], first_end.filter_dict), engine=engine, cache_store=cache_store) + second_leaf_nodes = _connected_join_cached_node_filter(base_graph, nodes_source, cast(Optional[dict], second_end.filter_dict), engine=engine, cache_store=cache_store) + if shared_ids is None or first_leaf_ids is None or second_leaf_ids is None: + return None + else: + shared_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], first_start.filter_dict)) + shared_nodes = filter_by_dict_polars(shared_nodes, cast(Optional[dict], second_start.filter_dict)) + first_leaf_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], first_end.filter_dict)) + second_leaf_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], second_end.filter_dict)) + if residual_map: + _frames = _connected_join_filter_node_frames_by_residuals( + base_graph, + residual_map, + {shared_alias: shared_nodes, first_end_alias: first_leaf_nodes, second_end_alias: second_leaf_nodes}, + node_col, + engine=engine, + ) + shared_nodes, first_leaf_nodes, second_leaf_nodes = _frames[shared_alias], _frames[first_end_alias], _frames[second_end_alias] + shared_ids = shared_nodes.select(node_col).unique() + first_leaf_ids = first_leaf_nodes.select(node_col).unique() + second_leaf_ids = second_leaf_nodes.select(node_col).unique() + first_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], first_edge.edge_match), engine=engine, cache_store=cache_store) + second_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], second_edge.edge_match), engine=engine, cache_store=cache_store) + + for _, prop in group_prop_refs: + if prop not in second_leaf_nodes.columns: + return None + + def singleton_id(ids: Any) -> Any: + if not isinstance(ids, pl.DataFrame) or len(ids) != 1: + return _CACHE_MISSING + return ids.get_column(node_col)[0] + + first_leaf_singleton = singleton_id(first_leaf_ids) + second_leaf_singleton = singleton_id(second_leaf_ids) + + cached_left_counts: Optional[DataFrameT] = None + if first_leaf_singleton is not _CACHE_MISSING and not residual_map: + shared_ids_for_left = shared_ids.rename({node_col: shared_alias}) if node_col != shared_alias else shared_ids + cached_left_counts = _connected_join_cached_first_arm_shared_counts( + base_graph, + nodes_source, + edges, + first_edges, + shared_ids_for_left, + cast(Optional[dict], first_edge.edge_match), + cast(Optional[dict], first_start.filter_dict), + cast(Optional[dict], second_start.filter_dict), + first_leaf_singleton, + shared_alias=shared_alias, + src_col=src_col, + dst_col=dst_col, + engine=engine, + cache_store=cache_store, + ) + # Unreachable under the current lowering: _connected_join_cached_first_arm_shared_counts + # only returns None when a filter/edge/dst cache key is uncacheable, but every value + # that reaches this cached polars path is cacheable (scalar equality pushes as scalars; + # comparisons/toLower/ranges lower to residuals routed off the cached path). Kept as a + # defensive fallback; excluded from coverage since no legitimate query reaches it. + if cached_left_counts is None: # pragma: no cover + cached_left_counts = _connected_join_cached_singleton_dst_source_counts( + base_graph, + edges, + first_edges, + cast(Optional[dict], first_edge.edge_match), + first_leaf_singleton, + src_col=src_col, + dst_col=dst_col, + engine=engine, + cache_store=cache_store, + ) + if cached_left_counts is not None: + if shared_alias in cached_left_counts.columns: + left_counts = cached_left_counts + else: + left_counts = ( + cached_left_counts + .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") + .rename({src_col: shared_alias}) + ) + elif first_leaf_singleton is _CACHE_MISSING: + left_edges = ( + first_edges + .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") + .join(first_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") + ) + left_counts = ( + left_edges + .group_by(src_col) + .len("__left_count__") + .rename({src_col: shared_alias}) + ) + else: + left_edges = ( + first_edges + .filter(pl.col(dst_col) == first_leaf_singleton) + .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") + ) + left_counts = ( + left_edges + .group_by(src_col) + .len("__left_count__") + .rename({src_col: shared_alias}) + ) + cached_right_rows = None if residual_map else _connected_join_cached_second_arm_group_rows( + base_graph, + nodes_source, + edges, + second_edges, + shared_ids, + second_leaf_ids, + second_leaf_nodes, + cast(Optional[dict], first_start.filter_dict), + cast(Optional[dict], second_start.filter_dict), + cast(Optional[dict], second_end.filter_dict), + cast(Optional[dict], second_edge.edge_match), + second_leaf_singleton, + group_prop_refs, + output_group_keys, + node_col=node_col, + src_col=src_col, + dst_col=dst_col, + engine=engine, + cache_store=cache_store, + ) + if cached_right_rows is not None: + right_rows = cached_right_rows if src_col == shared_alias else cached_right_rows.rename({src_col: shared_alias}) + else: + if second_leaf_singleton is _CACHE_MISSING: + right_base = ( + second_edges + .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") + .join(second_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") + ) + else: + right_base = ( + second_edges + .filter(pl.col(dst_col) == second_leaf_singleton) + .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") + ) + if group_prop_refs: + lookup_key = "__gfql_fast_second_leaf_id__" + lookup_exprs = [pl.col(node_col).alias(lookup_key)] + [pl.col(prop).alias(out_col) for out_col, prop in group_prop_refs] + # Dedup by node identity to match the pandas drop_duplicates(subset=[node_col]); + # a duplicate node row would otherwise multiply the join and over-count. + second_lookup = second_leaf_nodes.select(lookup_exprs).unique(subset=[lookup_key]) + right_base = right_base.join(second_lookup, left_on=dst_col, right_on=lookup_key, how="inner") + right_rows = right_base.select([pl.col(src_col).alias(shared_alias)] + [pl.col(key) for key in output_group_keys]) + if not output_group_keys and len(left_counts) > 0 and bool(left_counts.select((pl.col("__left_count__") == 1).all()).item()): + matched_rows = right_rows.join(left_counts.select(shared_alias), on=shared_alias, how="semi") + out_df = matched_rows.select(pl.len().cast(pl.Int64).alias(agg_alias)) + else: + joined = right_rows.join(left_counts, on=shared_alias, how="inner") + if len(joined) == 0: + return cast(DataFrameT, joined.select([])) + if output_group_keys: + out_df = joined.group_by(output_group_keys, maintain_order=True).agg(pl.col("__left_count__").sum().cast(pl.Int64).alias(agg_alias)) + else: + out_df = joined.select(pl.col("__left_count__").sum().cast(pl.Int64).alias(agg_alias)) + if order_keys: + # openCypher orders NULL as the largest value: ASC -> nulls last, DESC -> nulls + # first. Polars defaults to nulls-first, which flips WHICH ROW an ORDER BY ... LIMIT + # returns, so pin nulls_last per key. + out_df = out_df.sort( + [key for key, _ in order_keys], + descending=[desc for _, desc in order_keys], + nulls_last=[not desc for _, desc in order_keys], + ) + if limit_value is not None: + out_df = out_df.head(limit_value) + if select_items is not None: + out_df = out_df.select([pl.col(src).alias(dst) for src, dst in select_items]) + return cast(DataFrameT, out_df) + + filter_engine = EngineAbstract(engine.value) + shared_nodes = filter_by_dict(nodes, cast(Optional[dict], first_start.filter_dict), engine=filter_engine) + shared_nodes = filter_by_dict(shared_nodes, cast(Optional[dict], second_start.filter_dict), engine=filter_engine) + first_leaf_nodes = filter_by_dict(nodes, cast(Optional[dict], first_end.filter_dict), engine=filter_engine) + second_leaf_nodes = filter_by_dict(nodes, cast(Optional[dict], second_end.filter_dict), engine=filter_engine) + if residual_map: + _frames = _connected_join_filter_node_frames_by_residuals( + base_graph, + residual_map, + {shared_alias: shared_nodes, first_end_alias: first_leaf_nodes, second_end_alias: second_leaf_nodes}, + node_col, + engine=engine, + ) + shared_nodes, first_leaf_nodes, second_leaf_nodes = _frames[shared_alias], _frames[first_end_alias], _frames[second_end_alias] + first_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], first_edge.edge_match), engine=engine, cache_store=cache_store) + second_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], second_edge.edge_match), engine=engine, cache_store=cache_store) + + shared_ids = shared_nodes[node_col].drop_duplicates() + first_leaf_ids = first_leaf_nodes[node_col].drop_duplicates() + second_leaf_ids = second_leaf_nodes[node_col].drop_duplicates() + for _, prop in group_prop_refs: + if prop not in second_leaf_nodes.columns: + return None + + left_edges = first_edges[first_edges[src_col].isin(shared_ids) & first_edges[dst_col].isin(first_leaf_ids)] + left_counts = left_edges.groupby(src_col, sort=False).size().reset_index(name="__left_count__").rename(columns={src_col: shared_alias}) + right_base = second_edges[second_edges[src_col].isin(shared_ids) & second_edges[dst_col].isin(second_leaf_ids)] + if group_prop_refs: + lookup_key = "__gfql_fast_second_leaf_id__" + prop_cols = [] + for _, prop in group_prop_refs: + if prop not in prop_cols: + prop_cols.append(prop) + second_lookup = second_leaf_nodes[[node_col] + prop_cols].drop_duplicates(subset=[node_col]).rename(columns={node_col: lookup_key}) + for out_col, prop in group_prop_refs: + second_lookup[out_col] = second_lookup[prop] + right_base = right_base.merge(second_lookup[[lookup_key] + output_group_keys], left_on=dst_col, right_on=lookup_key, how="inner") + right_rows = right_base[[src_col] + output_group_keys].rename(columns={src_col: shared_alias}) + joined = right_rows.merge(left_counts, on=shared_alias, how="inner") + if len(joined) == 0: + return df_cons(engine)() + if output_group_keys: + out_df = joined.groupby(output_group_keys, sort=False, dropna=False)["__left_count__"].sum().reset_index(name=agg_alias) + else: + out_df = pd.DataFrame({agg_alias: [int(joined["__left_count__"].sum())]}) + if order_keys: + out_df = cast(DataFrameT, out_df.sort_values(by=[key for key, _ in order_keys], ascending=[not desc for _, desc in order_keys])) + if limit_value is not None: + out_df = cast(DataFrameT, out_df.head(limit_value)) + if select_items is not None: + out_df = out_df[[src for src, _ in select_items]].rename(columns={src: dst for src, dst in select_items}) + return df_to_engine(out_df.reset_index(drop=True), engine) + + +def _connected_join_two_star_fast_rows( + base_graph: Plottable, + plan: ConnectedMatchJoinPlan, + *, + engine: Engine, + cache_store: Optional[Dict[str, Any]] = None, +) -> Optional[DataFrameT]: + """Fast rows for T1 two-star connected joins. + + Supported shape: ``(p)-[r1]->(i), (p)-[r2]->(c)`` where both arms are fixed + forward one-hop patterns, share the same start node alias, have no residual + per-arm row filters, and downstream projections need at most properties from + the second leaf alias. The returned frame preserves row multiplicity, then + the normal row pipeline handles RETURN/GROUP/ORDER/LIMIT. + """ + # Per-execution cache scope (see _connected_join_two_star_fast_grouped_count); never + # persisted on the caller's Plottable. + if cache_store is None: + cache_store = {} + if len(plan.pattern_chains) != 2 or len(plan.pattern_shared_node_aliases) != 1: + return None + if len(plan.pattern_shared_node_aliases[0]) != 1: + return None + shared_alias = plan.pattern_shared_node_aliases[0][0] + + parsed = [] + for pattern_chain in plan.pattern_chains: + if pattern_chain.where: + return None + ops = list(pattern_chain.chain) + if len(ops) != 3: + return None + start_op, edge_op, end_op = ops + if not isinstance(start_op, ASTNode) or not isinstance(edge_op, ASTEdge) or not isinstance(end_op, ASTNode): + return None + if getattr(start_op, "query", None) is not None or getattr(end_op, "query", None) is not None: + return None + if not _is_connected_fast_single_hop(edge_op): + return None + start_alias = getattr(start_op, "_name", None) + end_alias = getattr(end_op, "_name", None) + if start_alias != shared_alias or not isinstance(end_alias, str): + return None + parsed.append((start_op, edge_op, end_op, end_alias)) + + first_start, first_edge, first_end, first_end_alias = parsed[0] + second_start, second_edge, second_end, second_end_alias = parsed[1] + if first_end_alias == second_end_alias: + return None + + attach_aliases = plan.pattern_attach_prop_aliases + if not attach_aliases or len(attach_aliases) != 2: + return None + normalized_attach_aliases: List[Tuple[str, ...]] = [] + for aliases in attach_aliases: + if aliases is None: + return None + normalized_attach_aliases.append(aliases) + needed_attach_aliases = {alias for aliases in normalized_attach_aliases for alias in aliases} + if any(alias != second_end_alias for alias in needed_attach_aliases): + return None + second_props_needed = _connected_join_post_property_columns(plan, second_end_alias) + # A bare aggregate over the second leaf (count(c) / count(DISTINCT c)) lowers `c` to its + # identity column c.__gfql_node_id__, which is NOT a materializable node property here -- + # fast_rows would drop it and post_join's count(c.__gfql_node_id__) would dereference a + # missing column. Decline to the (correct) slow path in that case. + if NODE_IDENTITY_COLUMN in second_props_needed: + return None + if second_end_alias in needed_attach_aliases and not second_props_needed: + return None + + nodes_obj = getattr(base_graph, "_nodes", None) + edges_obj = getattr(base_graph, "_edges", None) + node_col = getattr(base_graph, "_node", None) + src_col = getattr(base_graph, "_source", None) + dst_col = getattr(base_graph, "_destination", None) + if nodes_obj is None or edges_obj is None or node_col is None or src_col is None or dst_col is None: + return None + node_col = str(node_col) + src_col = str(src_col) + dst_col = str(dst_col) + if node_col not in nodes_obj.columns or src_col not in edges_obj.columns or dst_col not in edges_obj.columns: + return None + + nodes = df_to_engine(cast(DataFrameT, nodes_obj), engine) + edges = cast(DataFrameT, edges_obj) + + if engine in POLARS_ENGINES: + import polars as pl + from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars + + shared_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], first_start.filter_dict)) + shared_nodes = filter_by_dict_polars(shared_nodes, cast(Optional[dict], second_start.filter_dict)) + first_leaf_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], first_end.filter_dict)) + second_leaf_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], second_end.filter_dict)) + first_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], first_edge.edge_match), engine=engine, cache_store=cache_store) + second_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], second_edge.edge_match), engine=engine, cache_store=cache_store) + + shared_ids = shared_nodes.select(node_col).unique() + first_leaf_ids = first_leaf_nodes.select(node_col).unique() + second_leaf_ids = second_leaf_nodes.select(node_col).unique() + second_props = [col for col in second_props_needed if col in second_leaf_nodes.columns] + + left_rows = ( + first_edges + .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") + .join(first_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") + .select(pl.col(src_col).alias(shared_alias)) + ) + right_base = ( + second_edges + .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") + .join(second_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") + ) + if second_props: + second_lookup_key = "__gfql_fast_second_leaf_id__" + # Dedup by node identity (matches pandas drop_duplicates(subset=[node_col])); a + # duplicate node row would otherwise multiply the join and inflate row multiplicity. + second_lookup = second_leaf_nodes.select([node_col] + second_props).unique(subset=[node_col]).rename({node_col: second_lookup_key}) + second_lookup = second_lookup.rename({col: f"{second_end_alias}.{col}" for col in second_props}) + right_base = right_base.join(second_lookup, left_on=dst_col, right_on=second_lookup_key, how="inner") + right_select = [pl.col(src_col).alias(shared_alias)] + [pl.col(f"{second_end_alias}.{col}") for col in second_props] + right_rows = right_base.select(right_select) + return cast(DataFrameT, left_rows.join(right_rows, on=shared_alias, how="inner")) + + filter_engine = EngineAbstract(engine.value) + shared_nodes = filter_by_dict(nodes, cast(Optional[dict], first_start.filter_dict), engine=filter_engine) + shared_nodes = filter_by_dict(shared_nodes, cast(Optional[dict], second_start.filter_dict), engine=filter_engine) + first_leaf_nodes = filter_by_dict(nodes, cast(Optional[dict], first_end.filter_dict), engine=filter_engine) + second_leaf_nodes = filter_by_dict(nodes, cast(Optional[dict], second_end.filter_dict), engine=filter_engine) + first_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], first_edge.edge_match), engine=engine, cache_store=cache_store) + second_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], second_edge.edge_match), engine=engine, cache_store=cache_store) + + shared_ids = shared_nodes[node_col].drop_duplicates() + first_leaf_ids = first_leaf_nodes[node_col].drop_duplicates() + second_leaf_ids = second_leaf_nodes[node_col].drop_duplicates() + second_props = [col for col in second_props_needed if col in second_leaf_nodes.columns] + + left_edges = first_edges[first_edges[src_col].isin(shared_ids) & first_edges[dst_col].isin(first_leaf_ids)] + left_rows = left_edges[[src_col]].rename(columns={src_col: shared_alias}) + right_base = second_edges[second_edges[src_col].isin(shared_ids) & second_edges[dst_col].isin(second_leaf_ids)] + if second_props: + second_lookup_key = "__gfql_fast_second_leaf_id__" + second_lookup_cols = [node_col] + second_props + second_lookup = second_leaf_nodes[second_lookup_cols].drop_duplicates(subset=[node_col]).rename( + columns={ + node_col: second_lookup_key, + **{col: f"{second_end_alias}.{col}" for col in second_props}, + } + ) + right_base = right_base.merge(second_lookup, left_on=dst_col, right_on=second_lookup_key, how="inner") + right_cols = [src_col] + [f"{second_end_alias}.{col}" for col in second_props] + right_rows = right_base[right_cols].rename(columns={src_col: shared_alias}) + return cast(DataFrameT, left_rows.merge(right_rows, on=shared_alias, how="inner")) + + +def _filter_nodes_for_fast_count(nodes: DataFrameT, filter_dict: Optional[dict], *, engine: Engine) -> DataFrameT: + if engine in POLARS_ENGINES: + from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars + return cast(DataFrameT, filter_by_dict_polars(nodes, filter_dict)) + return filter_by_dict(nodes, filter_dict, engine=EngineAbstract(engine.value)) + + +def _two_hop_count_alias(chain: Chain) -> Optional[str]: + ops = list(chain.chain) + if len(ops) != 4 or not all(isinstance(op, ASTCall) for op in ops): + return None + rows_call, with_call, group_call, select_call = cast(Tuple[ASTCall, ASTCall, ASTCall, ASTCall], tuple(ops)) + if rows_call.function != "rows" or with_call.function != "with_" or group_call.function != "group_by" or select_call.function != "select": + return None + if rows_call.params.get("table") != "nodes": + return None + if with_call.params.get("items") != [("__cypher_group__", 1)]: + return None + aggs = group_call.params.get("aggregations") + if group_call.params.get("keys") != ["__cypher_group__"] or not isinstance(aggs, list) or len(aggs) != 1: + return None + agg = aggs[0] + if not isinstance(agg, (tuple, list)) or len(agg) != 2 or str(agg[1]).lower() != "count": + return None + alias = str(agg[0]) + if select_call.params.get("items") != [(alias, alias)]: + return None + return alias + + +def _two_hop_count_binding_ops(chain: Chain) -> Optional[Tuple[ASTNode, ASTEdge, ASTNode, ASTEdge, ASTNode]]: + if not chain.chain or not isinstance(chain.chain[0], ASTCall): + return None + rows_call = cast(ASTCall, chain.chain[0]) + binding_ops = rows_call.params.get("binding_ops") + if not isinstance(binding_ops, list) or len(binding_ops) != 5: + return None + from graphistry.compute.ast import from_json as ast_from_json + ops = [ast_from_json(op_json, validate=False) for op_json in binding_ops] + n0, e0, n1, e1, n2 = ops + if not isinstance(n0, ASTNode) or not isinstance(e0, ASTEdge) or not isinstance(n1, ASTNode) or not isinstance(e1, ASTEdge) or not isinstance(n2, ASTNode): + return None + if not _is_connected_fast_single_hop(e0) or not _is_connected_fast_single_hop(e1): + return None + return n0, e0, n1, e1, n2 + + +def _property_ref(expr: Any, valid_aliases: Sequence[str]) -> Optional[Tuple[str, str]]: + if not isinstance(expr, str) or "." not in expr: + return None + alias, prop = expr.split(".", 1) + if alias not in valid_aliases or not prop: + return None + return alias, prop + + +def _execute_single_hop_grouped_aggregate_fast_path( + base_graph: Plottable, + chain: Chain, + *, + engine: Union[EngineAbstract, str], +) -> Optional[Plottable]: + ops = list(chain.chain) + if len(ops) not in (3, 4, 5) or not all(isinstance(op, ASTCall) for op in ops): + return None + rows_call = cast(ASTCall, ops[0]) + with_call = cast(ASTCall, ops[1]) + group_call = cast(ASTCall, ops[2]) + suffix = [cast(ASTCall, op) for op in ops[3:]] + if rows_call.function != "rows" or with_call.function != "with_" or group_call.function != "group_by": + return None + if rows_call.params.get("table") != "nodes" or with_call.params.get("extend") is not True: + return None + + order_call: Optional[ASTCall] = None + limit_call: Optional[ASTCall] = None + for op in suffix: + if op.function == "order_by" and order_call is None and limit_call is None: + order_call = op + elif op.function == "limit" and limit_call is None: + limit_call = op + else: + return None + + binding_ops = rows_call.params.get("binding_ops") + if not isinstance(binding_ops, list) or len(binding_ops) != 3: + return None + from graphistry.compute.ast import from_json as ast_from_json + parsed_ops = [ast_from_json(op_json, validate=False) for op_json in binding_ops] + start_op, edge_op, end_op = parsed_ops + if not isinstance(start_op, ASTNode) or not isinstance(edge_op, ASTEdge) or not isinstance(end_op, ASTNode): + return None + if not _is_connected_fast_single_hop(edge_op): + return None + start_alias = getattr(start_op, "_name", None) + end_alias = getattr(end_op, "_name", None) + if not isinstance(start_alias, str) or not isinstance(end_alias, str) or start_alias == end_alias: + return None + aliases = (start_alias, end_alias) + + with_items_raw = with_call.params.get("items") + if not isinstance(with_items_raw, list): + return None + with_items: Dict[str, Tuple[str, Optional[str]]] = {} + for item in with_items_raw: + if not isinstance(item, (tuple, list)) or len(item) != 2 or not isinstance(item[0], str): + return None + prop_ref = _property_ref(item[1], aliases) + ref: Tuple[str, Optional[str]] + if prop_ref is None: + if item[1] not in aliases: + return None + ref = (cast(str, item[1]), None) + else: + ref = prop_ref + with_items[item[0]] = ref + + group_keys_raw = group_call.params.get("keys") + if not isinstance(group_keys_raw, list) or not group_keys_raw or not all(isinstance(key, str) for key in group_keys_raw): + return None + group_keys = cast(List[str], group_keys_raw) + if not all(key in with_items and with_items[key][1] is not None for key in group_keys): + return None + + aggs_raw = group_call.params.get("aggregations") + if not isinstance(aggs_raw, list) or not aggs_raw: + return None + aggregations: List[Tuple[str, str, Optional[str]]] = [] + for agg in aggs_raw: + if not isinstance(agg, (tuple, list)) or len(agg) not in (2, 3): + return None + alias = str(agg[0]) + func = str(agg[1]).lower() + expr_alias = agg[2] if len(agg) == 3 else None + if func not in {"count", "avg", "sum", "min", "max"}: + return None + if expr_alias is not None and not isinstance(expr_alias, str): + return None + if expr_alias is not None and expr_alias not in with_items: + return None + if func != "count" and expr_alias is None: + return None + if expr_alias is not None and with_items[expr_alias][1] is None and func != "count": + return None + aggregations.append((alias, func, cast(Optional[str], expr_alias))) + + order_keys: List[Tuple[str, bool]] = [] + if order_call is not None: + raw_order = order_call.params.get("keys") + if not isinstance(raw_order, list): + return None + available = set(group_keys) | {alias for alias, _, _ in aggregations} + for item in raw_order: + if not isinstance(item, (tuple, list)) or len(item) != 2 or not isinstance(item[0], str): + return None + if item[0] not in available: + return None + direction = str(item[1]).lower() + if direction not in {"asc", "ascending", "desc", "descending"}: + return None + order_keys.append((item[0], direction in {"desc", "descending"})) + + limit_value: Optional[int] = None + if limit_call is not None: + raw_limit = limit_call.params.get("value") + if not isinstance(raw_limit, int) or raw_limit < 0: + return None + limit_value = raw_limit + + requested_engine = resolve_engine(cast(Any, engine), base_graph) + nodes_obj = getattr(base_graph, "_nodes", None) + edges_obj = getattr(base_graph, "_edges", None) + node_col = getattr(base_graph, "_node", None) + src_col = getattr(base_graph, "_source", None) + dst_col = getattr(base_graph, "_destination", None) + if nodes_obj is None or edges_obj is None or node_col is None or src_col is None or dst_col is None: + return None + node_col = str(node_col) + src_col = str(src_col) + dst_col = str(dst_col) + if node_col not in nodes_obj.columns or src_col not in edges_obj.columns or dst_col not in edges_obj.columns: + return None + + nodes = cast(DataFrameT, nodes_obj) + start_nodes = _connected_join_cached_node_filter(base_graph, nodes, cast(Optional[dict], start_op.filter_dict), engine=requested_engine) + end_nodes = _connected_join_cached_node_filter(base_graph, nodes, cast(Optional[dict], end_op.filter_dict), engine=requested_engine) + edges = _connected_join_cached_edge_filter(base_graph, cast(DataFrameT, edges_obj), cast(Optional[dict], edge_op.edge_match), engine=requested_engine) + + needed_by_alias: Dict[str, List[Tuple[str, str]]] = {start_alias: [], end_alias: []} + for out_col, ref in with_items.items(): + alias, prop = ref + if prop is not None: + needed_by_alias[alias].append((out_col, prop)) + + if requested_engine in POLARS_ENGINES: # pragma: no cover - polars-only, covered by polars lane + import polars as pl + start_ids = start_nodes.select(node_col).unique() + end_ids = end_nodes.select(node_col).unique() + work = ( + edges + .join(start_ids, left_on=src_col, right_on=node_col, how="semi") + .join(end_ids, left_on=dst_col, right_on=node_col, how="semi") + .select([src_col, dst_col]) + ) + + def join_props_polars(work_df: Any, alias: str, node_df: Any, edge_col: str) -> Any: + props = needed_by_alias.get(alias, []) + if not props: + return work_df + lookup_key = f"__gfql_t3_{alias}_id__" + exprs = [pl.col(node_col).alias(lookup_key)] + for out_col, prop in props: + if prop not in node_df.columns: + return None + exprs.append(pl.col(prop).alias(out_col)) + lookup = node_df.select(exprs) + return work_df.join(lookup, left_on=edge_col, right_on=lookup_key, how="inner") + + work = join_props_polars(work, start_alias, start_nodes, src_col) + if work is None: + return None + work = join_props_polars(work, end_alias, end_nodes, dst_col) + if work is None: + return None + agg_exprs = [] + for alias, func, expr_alias in aggregations: + if func == "count" and (expr_alias is None or with_items[expr_alias][1] is None): + agg_exprs.append(pl.len().alias(alias)) + elif func == "count" and expr_alias is not None: + agg_exprs.append(pl.col(expr_alias).count().alias(alias)) + elif func == "avg" and expr_alias is not None: + agg_exprs.append(pl.col(expr_alias).mean().alias(alias)) + elif func == "sum" and expr_alias is not None: + agg_exprs.append(pl.col(expr_alias).sum().alias(alias)) + elif func == "min" and expr_alias is not None: + agg_exprs.append(pl.col(expr_alias).min().alias(alias)) + elif func == "max" and expr_alias is not None: + agg_exprs.append(pl.col(expr_alias).max().alias(alias)) + else: + return None + out_nodes = work.group_by(group_keys, maintain_order=True).agg(agg_exprs) + if order_keys: + # openCypher orders NULL as the largest value (ASC -> nulls last, DESC -> nulls + # first). Polars defaults nulls-first, which flips WHICH ROW ORDER BY ... LIMIT + # returns, so pin nulls_last per key. + out_nodes = out_nodes.sort( + [key for key, _ in order_keys], + descending=[desc for _, desc in order_keys], + nulls_last=[not desc for _, desc in order_keys], + ) + if limit_value is not None: + out_nodes = out_nodes.head(limit_value) + out_df = cast(DataFrameT, out_nodes) + else: + start_ids = start_nodes[node_col].drop_duplicates() + end_ids = end_nodes[node_col].drop_duplicates() + work = edges[edges[src_col].isin(start_ids) & edges[dst_col].isin(end_ids)][[src_col, dst_col]] + + def join_props_df(work_df: DataFrameT, alias: str, node_df: DataFrameT, edge_col: str) -> Optional[DataFrameT]: + props = needed_by_alias.get(alias, []) + if not props: + return work_df + prop_cols = [] + for _, prop in props: + if prop not in node_df.columns: + return None + if prop != node_col and prop not in prop_cols: + prop_cols.append(prop) + lookup_key = f"__gfql_t3_{alias}_id__" + lookup = node_df[[node_col] + prop_cols].drop_duplicates(subset=[node_col]).copy() + for out_col, prop in props: + lookup[out_col] = lookup[prop] + lookup = lookup.rename(columns={node_col: lookup_key}) + return cast(DataFrameT, work_df.merge(lookup, left_on=edge_col, right_on=lookup_key, how="inner")) + + work = cast(DataFrameT, work) + joined_start = join_props_df(work, start_alias, start_nodes, src_col) + if joined_start is None: + return None + joined_end = join_props_df(joined_start, end_alias, end_nodes, dst_col) + if joined_end is None: + return None + work = joined_end + try: + grouped = work.groupby(group_keys, sort=False, dropna=False) + except TypeError: + grouped = work.groupby(group_keys, sort=False) + out_df = grouped.size().reset_index(name="__gfql_group_size__")[group_keys] + for alias, func, expr_alias in aggregations: + if func == "count" and (expr_alias is None or with_items[expr_alias][1] is None): + agg_df = grouped.size().reset_index(name=alias) + elif func == "count" and expr_alias is not None: + agg_df = grouped[expr_alias].count().reset_index(name=alias) + elif func == "avg" and expr_alias is not None: + agg_df = grouped[expr_alias].mean().reset_index(name=alias) + elif func == "sum" and expr_alias is not None: + agg_df = grouped[expr_alias].sum().reset_index(name=alias) + elif func == "min" and expr_alias is not None: + agg_df = grouped[expr_alias].min().reset_index(name=alias) + elif func == "max" and expr_alias is not None: + agg_df = grouped[expr_alias].max().reset_index(name=alias) + else: + return None + out_df = cast(DataFrameT, out_df.merge(agg_df, on=group_keys, how="left", sort=False)) + if order_keys: + # openCypher orders NULL as the largest value (ASC -> nulls last, DESC -> nulls + # first), matching the polars branch's per-key nulls_last above. pandas/cuDF + # sort_values take a SCALAR na_position (can't be per-key), and cuDF has no stable + # sort (kind='stable' silently falls back to quicksort), so a per-key multi-pass would + # lose tie order on GPU. Instead express null placement with an explicit per-key + # null-indicator column and sort in ONE pass: a key sorted ascending=(not desc) with + # its indicator sorted the same way puts NULL at the correct (largest) end, no stable + # sort required. The default single sort_values used pandas' na_position='last' for + # every key, so DESC keys put NULL last, flipping which row an ORDER BY ... LIMIT keeps. + sort_cols: List[str] = [] + ascending: List[bool] = [] + helper_cols: List[str] = [] + for i, (key, desc) in enumerate(order_keys): + indicator = f"__gfql_nullrank_{i}__" + out_df[indicator] = out_df[key].isna() + sort_cols.extend([indicator, key]) + ascending.extend([not desc, not desc]) + helper_cols.append(indicator) + out_df = out_df.sort_values(by=sort_cols, ascending=ascending) + out_df = cast(DataFrameT, out_df.drop(columns=helper_cols)) + if limit_value is not None: + out_df = cast(DataFrameT, out_df.head(limit_value)) + out_df = df_to_engine(out_df.reset_index(drop=True), requested_engine) + + out = base_graph.bind() + out._nodes = out_df + out._edges = df_cons(requested_engine)() + return out + + +def _execute_two_hop_count_fast_path( + base_graph: Plottable, + chain: Chain, + *, + engine: Union[EngineAbstract, str], +) -> Optional[Plottable]: + alias = _two_hop_count_alias(chain) + if alias is None: + return None + ops = _two_hop_count_binding_ops(chain) + if ops is None: + return None + start_op, first_edge, middle_op, second_edge, end_op = ops + + requested_engine = resolve_engine(cast(Any, engine), base_graph) + nodes_obj = getattr(base_graph, "_nodes", None) + edges_obj = getattr(base_graph, "_edges", None) + node_col = getattr(base_graph, "_node", None) + src_col = getattr(base_graph, "_source", None) + dst_col = getattr(base_graph, "_destination", None) + if nodes_obj is None or edges_obj is None or node_col is None or src_col is None or dst_col is None: + return None + node_col = str(node_col) + src_col = str(src_col) + dst_col = str(dst_col) + if node_col not in nodes_obj.columns or src_col not in edges_obj.columns or dst_col not in edges_obj.columns: + return None + + nodes = cast(DataFrameT, nodes_obj) + start_nodes = _connected_join_cached_node_filter(base_graph, nodes, cast(Optional[dict], start_op.filter_dict), engine=requested_engine) + middle_nodes = ( + start_nodes + if middle_op.filter_dict == start_op.filter_dict + else _connected_join_cached_node_filter(base_graph, nodes, cast(Optional[dict], middle_op.filter_dict), engine=requested_engine) + ) + end_nodes = ( + middle_nodes + if end_op.filter_dict == middle_op.filter_dict + else start_nodes + if end_op.filter_dict == start_op.filter_dict + else _connected_join_cached_node_filter(base_graph, nodes, cast(Optional[dict], end_op.filter_dict), engine=requested_engine) + ) + first_edges = _connected_join_cached_edge_filter(base_graph, cast(DataFrameT, edges_obj), cast(Optional[dict], first_edge.edge_match), engine=requested_engine) + reuse_single_edge_domain = ( + start_op.filter_dict == middle_op.filter_dict == end_op.filter_dict + and first_edge.edge_match == second_edge.edge_match + ) + second_edges = ( + first_edges + if first_edge.edge_match == second_edge.edge_match + else _connected_join_cached_edge_filter(base_graph, cast(DataFrameT, edges_obj), cast(Optional[dict], second_edge.edge_match), engine=requested_engine) + ) + + if requested_engine in POLARS_ENGINES: + import polars as pl + if reuse_single_edge_domain: + cached_counts = _two_hop_cached_equal_domain_degree_counts( + base_graph, + nodes, + cast(DataFrameT, edges_obj), + start_nodes, + first_edges, + node_match=cast(Optional[dict], start_op.filter_dict), + edge_match=cast(Optional[dict], first_edge.edge_match), + node_col=node_col, + src_col=src_col, + dst_col=dst_col, + engine=requested_engine, + ) + if cached_counts is None: + domain_ids = start_nodes.select(node_col).unique() + domain_edges = ( + first_edges + .join(domain_ids, left_on=src_col, right_on=node_col, how="semi") + .join(domain_ids, left_on=dst_col, right_on=node_col, how="semi") + ) + in_counts = domain_edges.group_by(dst_col).len("__in_count__") + out_counts = domain_edges.group_by(src_col).len("__out_count__") + else: + in_counts, out_counts = cached_counts + else: + start_ids = start_nodes.select(node_col).unique() + middle_ids = middle_nodes.select(node_col).unique() + end_ids = end_nodes.select(node_col).unique() + in_counts = ( + first_edges + .join(start_ids, left_on=src_col, right_on=node_col, how="semi") + .join(middle_ids, left_on=dst_col, right_on=node_col, how="semi") + .group_by(dst_col) + .len("__in_count__") + ) + out_counts = ( + second_edges + .join(middle_ids, left_on=src_col, right_on=node_col, how="semi") + .join(end_ids, left_on=dst_col, right_on=node_col, how="semi") + .group_by(src_col) + .len("__out_count__") + ) + total_df = ( + in_counts + .join(out_counts, left_on=dst_col, right_on=src_col, how="inner") + .select((pl.col("__in_count__") * pl.col("__out_count__")).sum().fill_null(0).cast(pl.Int64).alias(alias)) + ) + out_nodes = cast(DataFrameT, total_df) + else: + if reuse_single_edge_domain: + cached_counts = _two_hop_cached_equal_domain_degree_counts( + base_graph, + nodes, + cast(DataFrameT, edges_obj), + start_nodes, + first_edges, + node_match=cast(Optional[dict], start_op.filter_dict), + edge_match=cast(Optional[dict], first_edge.edge_match), + node_col=node_col, + src_col=src_col, + dst_col=dst_col, + engine=requested_engine, + ) + if cached_counts is None: + domain_ids = start_nodes[node_col].drop_duplicates() + domain_edges = first_edges[first_edges[src_col].isin(domain_ids) & first_edges[dst_col].isin(domain_ids)] + in_counts = domain_edges.groupby(dst_col, sort=False).size().reset_index(name="__in_count__") + out_counts = domain_edges.groupby(src_col, sort=False).size().reset_index(name="__out_count__") + else: + in_counts, out_counts = cached_counts + else: + start_ids = start_nodes[node_col].drop_duplicates() + middle_ids = middle_nodes[node_col].drop_duplicates() + end_ids = end_nodes[node_col].drop_duplicates() + in_edges = first_edges[first_edges[src_col].isin(start_ids) & first_edges[dst_col].isin(middle_ids)] + out_edges = second_edges[second_edges[src_col].isin(middle_ids) & second_edges[dst_col].isin(end_ids)] + in_counts = in_edges.groupby(dst_col, sort=False).size().reset_index(name="__in_count__") + out_counts = out_edges.groupby(src_col, sort=False).size().reset_index(name="__out_count__") + joined = in_counts.merge(out_counts, left_on=dst_col, right_on=src_col, how="inner") + total = int((joined["__in_count__"] * joined["__out_count__"]).sum()) if len(joined) else 0 + out_nodes = df_to_engine(pd.DataFrame({alias: [total]}), requested_engine) + + out = base_graph.bind() + out._nodes = out_nodes + out._edges = df_cons(requested_engine)() + return out diff --git a/graphistry/compute/gfql_unified.py b/graphistry/compute/gfql_unified.py index 3aaf151bd1..d043698044 100644 --- a/graphistry/compute/gfql_unified.py +++ b/graphistry/compute/gfql_unified.py @@ -476,1113 +476,14 @@ def _optional_arm_start_nodes( -def _is_connected_fast_single_hop(edge_op: ASTEdge) -> bool: - return ( - getattr(edge_op, "direction", None) == "forward" - and getattr(edge_op, "hops", None) in (None, 1) - and getattr(edge_op, "min_hops", None) is None - and getattr(edge_op, "max_hops", None) is None - and getattr(edge_op, "output_min_hops", None) is None - and getattr(edge_op, "output_max_hops", None) is None - and getattr(edge_op, "label_node_hops", None) is None - and getattr(edge_op, "label_edge_hops", None) is None - and not bool(getattr(edge_op, "label_seeds", False)) - and not bool(getattr(edge_op, "to_fixed_point", False)) - and getattr(edge_op, "source_node_match", None) is None - and getattr(edge_op, "destination_node_match", None) is None - and getattr(edge_op, "source_node_query", None) is None - and getattr(edge_op, "destination_node_query", None) is None - and getattr(edge_op, "edge_query", None) is None - and getattr(edge_op, "_name", None) is None - and not bool(getattr(edge_op, "include_zero_hop_seed", False)) - ) - - - - -_CACHE_MISSING = object() - - -def _connected_join_filter_value_cache_key(value: Any) -> Optional[Tuple[str, str]]: - if isinstance(value, bool): - return "bool", "true" if value else "false" - if isinstance(value, str): - return "str", value - if isinstance(value, int) and not isinstance(value, bool): - return "int", str(value) - if isinstance(value, float): - return "float", repr(value) - if value is None: - return "none", "" - - predicate_value = getattr(value, "val", _CACHE_MISSING) - if predicate_value is not _CACHE_MISSING: - scalar_key = _connected_join_filter_value_cache_key(predicate_value) - if scalar_key is None: - return None - return f"{type(value).__module__}.{type(value).__qualname__}", f"{scalar_key[0]}:{scalar_key[1]}" - - regex_pattern = getattr(value, "pat", _CACHE_MISSING) - regex_case = getattr(value, "case", _CACHE_MISSING) - regex_flags = getattr(value, "flags", _CACHE_MISSING) - regex_na = getattr(value, "na", _CACHE_MISSING) - if ( - regex_pattern is not _CACHE_MISSING - and regex_case is not _CACHE_MISSING - and regex_flags is not _CACHE_MISSING - and regex_na is not _CACHE_MISSING - ): - pattern_key = _connected_join_filter_value_cache_key(regex_pattern) - case_key = _connected_join_filter_value_cache_key(regex_case) - flags_key = _connected_join_filter_value_cache_key(regex_flags) - na_key = _connected_join_filter_value_cache_key(regex_na) - if pattern_key is None or case_key is None or flags_key is None or na_key is None: - return None - return ( - f"{type(value).__module__}.{type(value).__qualname__}", - f"pat={pattern_key[0]}:{pattern_key[1]}|case={case_key[0]}:{case_key[1]}|flags={flags_key[0]}:{flags_key[1]}|na={na_key[0]}:{na_key[1]}", - ) - - predicates = getattr(value, "predicates", _CACHE_MISSING) - if predicates is not _CACHE_MISSING and isinstance(predicates, (list, tuple)): - child_keys = [] - for predicate in predicates: - child_key = _connected_join_filter_value_cache_key(predicate) - if child_key is None: - return None - child_keys.append(f"{child_key[0]}:{child_key[1]}") - return f"{type(value).__module__}.{type(value).__qualname__}", "|".join(child_keys) - - return None - - -def _connected_join_simple_filter_cache_key(filter_dict: Optional[dict]) -> Optional[Tuple[Tuple[str, str, str], ...]]: - if not filter_dict: - return () - items: List[Tuple[str, str, str]] = [] - for key, value in filter_dict.items(): - if not isinstance(key, str): - return None - value_key = _connected_join_filter_value_cache_key(value) - if value_key is None: - return None - items.append((key, value_key[0], value_key[1])) - return tuple(sorted(items)) - - -def _connected_join_cached_node_filter( - base_graph: Plottable, - nodes_obj: DataFrameT, - node_match: Optional[dict], - *, - engine: Engine, - cache_store: Optional[Dict[str, Any]] = None, -) -> DataFrameT: - cache_key = _connected_join_simple_filter_cache_key(node_match) - if cache_key is None: - nodes = df_to_engine(nodes_obj, engine) - if engine in POLARS_ENGINES: - from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars - return cast(DataFrameT, filter_by_dict_polars(nodes, node_match)) - return filter_by_dict(nodes, node_match, engine=EngineAbstract(engine.value)) - - cache_attr = "_gfql_connected_join_node_filter_cache" - # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's - # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale - # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. - cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None - full_key = (id(nodes_obj), engine.value, cache_key) - if cache is not None and full_key in cache: - return cast(DataFrameT, cache[full_key]) - - nodes = df_to_engine(nodes_obj, engine) - if engine in POLARS_ENGINES: - from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars - filtered = cast(DataFrameT, filter_by_dict_polars(nodes, node_match)) - else: - filtered = filter_by_dict(nodes, node_match, engine=EngineAbstract(engine.value)) - if cache is not None: - cache[full_key] = filtered - return cast(DataFrameT, filtered) - - -def _connected_join_cached_node_ids( - base_graph: Plottable, - nodes_obj: DataFrameT, - node_matches: Sequence[Optional[dict]], - *, - node_col: str, - engine: Engine, - cache_store: Optional[Dict[str, Any]] = None, -) -> Optional[DataFrameT]: - if engine not in POLARS_ENGINES: - return None - - match_keys: List[Tuple[Tuple[str, str, str], ...]] = [] - for node_match in node_matches: - match_key = _connected_join_simple_filter_cache_key(node_match) - if match_key is None: - return None - match_keys.append(match_key) - - cache_attr = "_gfql_connected_join_node_ids_cache" - # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's - # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale - # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. - cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None - full_key = (id(nodes_obj), engine.value, node_col, tuple(match_keys)) - if cache is not None and full_key in cache: - return cast(DataFrameT, cache[full_key]) - - from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars - - nodes = df_to_engine(nodes_obj, engine) - filtered = nodes - for node_match in node_matches: - filtered = cast(DataFrameT, filter_by_dict_polars(filtered, node_match)) - ids = cast(DataFrameT, filtered.select(node_col).unique()) - if cache is not None: - cache[full_key] = ids - return ids - - -def _connected_join_cached_edge_filter( - base_graph: Plottable, - edges_obj: DataFrameT, - edge_match: Optional[dict], - *, - engine: Engine, - cache_store: Optional[Dict[str, Any]] = None, -) -> DataFrameT: - cache_key = _connected_join_simple_filter_cache_key(edge_match) - if cache_key is None: - edges = df_to_engine(edges_obj, engine) - if engine in POLARS_ENGINES: - from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars - return cast(DataFrameT, filter_by_dict_polars(edges, edge_match)) - return filter_by_dict(edges, edge_match, engine=EngineAbstract(engine.value)) - - cache_attr = "_gfql_connected_join_edge_filter_cache" - # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's - # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale - # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. - cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None - full_key = (id(edges_obj), engine.value, cache_key) - if cache is not None and full_key in cache: - return cast(DataFrameT, cache[full_key]) - - edges = df_to_engine(edges_obj, engine) - if engine in POLARS_ENGINES: - from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars - filtered = cast(DataFrameT, filter_by_dict_polars(edges, edge_match)) - else: - filtered = filter_by_dict(edges, edge_match, engine=EngineAbstract(engine.value)) - if cache is not None: - cache[full_key] = filtered - return cast(DataFrameT, filtered) - - -def _connected_join_cached_singleton_dst_source_counts( # pragma: no cover - base_graph: Plottable, - edges_obj: DataFrameT, - edge_domain: DataFrameT, - edge_match: Optional[dict], - singleton_dst: Any, - *, - src_col: str, - dst_col: str, - engine: Engine, - cache_store: Optional[Dict[str, Any]] = None, -) -> Optional[DataFrameT]: - edge_key = _connected_join_simple_filter_cache_key(edge_match) - dst_key = _connected_join_filter_value_cache_key(singleton_dst) - if edge_key is None or dst_key is None: - return None - - cache_attr = "_gfql_connected_join_singleton_dst_source_counts_cache" - # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's - # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale - # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. - cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None - full_key = (id(edges_obj), engine.value, src_col, dst_col, edge_key, dst_key) - if cache is not None and full_key in cache: - return cast(DataFrameT, cache[full_key]) - - if engine in POLARS_ENGINES: - import polars as pl - counts = cast( - DataFrameT, - edge_domain - .filter(pl.col(dst_col) == singleton_dst) - .group_by(src_col) - .len("__left_count__"), - ) - else: - filtered = edge_domain[edge_domain[dst_col] == singleton_dst] - counts = cast(DataFrameT, filtered.groupby(src_col, sort=False).size().reset_index(name="__left_count__")) - - if cache is not None: - cache[full_key] = counts - return counts - - -def _connected_join_cached_first_arm_shared_counts( - base_graph: Plottable, - nodes_obj: DataFrameT, - edges_obj: DataFrameT, - first_edges: DataFrameT, - shared_ids: DataFrameT, - first_edge_match: Optional[dict], - first_start_match: Optional[dict], - second_start_match: Optional[dict], - singleton_dst: Any, - *, - shared_alias: str, - src_col: str, - dst_col: str, - engine: Engine, - cache_store: Optional[Dict[str, Any]] = None, -) -> Optional[DataFrameT]: - if engine not in POLARS_ENGINES: - return None - - edge_key = _connected_join_simple_filter_cache_key(first_edge_match) - first_start_key = _connected_join_simple_filter_cache_key(first_start_match) - second_start_key = _connected_join_simple_filter_cache_key(second_start_match) - dst_key = _connected_join_filter_value_cache_key(singleton_dst) - if edge_key is None or first_start_key is None or second_start_key is None or dst_key is None: - return None - - cache_attr = "_gfql_connected_join_first_arm_shared_counts_cache" - # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's - # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale - # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. - cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None - full_key = ( - id(nodes_obj), - id(edges_obj), - engine.value, - shared_alias, - src_col, - dst_col, - edge_key, - first_start_key, - second_start_key, - dst_key, - ) - if cache is not None and full_key in cache: - return cast(DataFrameT, cache[full_key]) - - import polars as pl - - counts = ( - first_edges - .filter(pl.col(dst_col) == singleton_dst) - .join(shared_ids, left_on=src_col, right_on=shared_alias, how="semi") - .group_by(src_col) - .len("__left_count__") - .rename({src_col: shared_alias}) - ) - if cache is not None: - cache[full_key] = counts - return cast(DataFrameT, counts) - - -def _connected_join_cached_second_arm_group_rows( - base_graph: Plottable, - nodes_obj: DataFrameT, - edges_obj: DataFrameT, - second_edges: DataFrameT, - shared_ids: DataFrameT, - second_leaf_ids: DataFrameT, - second_leaf_nodes: DataFrameT, - first_start_match: Optional[dict], - second_start_match: Optional[dict], - second_leaf_match: Optional[dict], - second_edge_match: Optional[dict], - second_leaf_singleton: Any, - group_prop_refs: List[Tuple[str, str]], - output_group_keys: List[str], - *, - node_col: str, - src_col: str, - dst_col: str, - engine: Engine, - cache_store: Optional[Dict[str, Any]] = None, -) -> Optional[DataFrameT]: - if engine not in POLARS_ENGINES: - return None - - first_start_key = _connected_join_simple_filter_cache_key(first_start_match) - second_start_key = _connected_join_simple_filter_cache_key(second_start_match) - second_leaf_key = _connected_join_simple_filter_cache_key(second_leaf_match) - second_edge_key = _connected_join_simple_filter_cache_key(second_edge_match) - if first_start_key is None or second_start_key is None or second_leaf_key is None or second_edge_key is None: - return None - - prop_key = tuple((str(out_col), str(prop)) for out_col, prop in group_prop_refs) - output_key = tuple(str(key) for key in output_group_keys) - cache_attr = "_gfql_connected_join_second_arm_group_rows_cache" - # Per-execution cache only (threaded via cache_store); NEVER setattr onto the caller's - # Plottable -- that leaked results across gfql() calls keyed by id(), returning stale - # answers after an in-place edge/node mutation (BLOCKER 1). None => no caching. - cache = cache_store.setdefault(cache_attr, {}) if cache_store is not None else None - full_key = ( - id(nodes_obj), - id(edges_obj), - engine.value, - node_col, - src_col, - dst_col, - first_start_key, - second_start_key, - second_leaf_key, - second_edge_key, - prop_key, - output_key, - ) - if cache is not None and full_key in cache: - return cast(DataFrameT, cache[full_key]) - - import polars as pl - - if second_leaf_singleton is _CACHE_MISSING: - right_base = ( - second_edges - .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") - .join(second_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") - ) - else: - right_base = ( - second_edges - .filter(pl.col(dst_col) == second_leaf_singleton) - .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") - ) - if group_prop_refs: - lookup_key = "__gfql_fast_second_leaf_id__" - lookup_exprs = [pl.col(node_col).alias(lookup_key)] + [pl.col(prop).alias(out_col) for out_col, prop in group_prop_refs] - # Dedup by node identity (matches pandas drop_duplicates(subset=[node_col])); a - # duplicate node row would otherwise multiply the join and over-count. - second_lookup = second_leaf_nodes.select(lookup_exprs).unique(subset=[lookup_key]) - right_base = right_base.join(second_lookup, left_on=dst_col, right_on=lookup_key, how="inner") - right_rows = right_base.select([pl.col(src_col)] + [pl.col(key) for key in output_group_keys]) - if cache is not None: - cache[full_key] = right_rows - return cast(DataFrameT, right_rows) - - -def _property_ref(expr: Any, valid_aliases: Sequence[str]) -> Optional[Tuple[str, str]]: - if not isinstance(expr, str) or "." not in expr: - return None - alias, prop = expr.split(".", 1) - if alias not in valid_aliases or not prop: - return None - return alias, prop - - -def _connected_join_post_property_columns(plan: ConnectedMatchJoinPlan, alias: str) -> List[str]: - prefix = f"{alias}." - out: List[str] = [] - - def walk(value: Any) -> None: - if isinstance(value, str): - if value.startswith(prefix): - prop = value[len(prefix):] - if prop and prop not in out: - out.append(prop) - return - if isinstance(value, Mapping): - for item in value.values(): - walk(item) - return - if isinstance(value, (list, tuple)): - for item in value: - walk(item) - - for op in plan.post_join_chain.chain: - if isinstance(op, ASTCall): - walk(op.params) - return out - - -def _connected_join_two_star_split_residuals( - plan: ConnectedMatchJoinPlan, - alias_targets: Mapping[str, ASTObject], - materialized_aliases: Set[str], -) -> Optional[Tuple[Dict[str, List[str]], "Chain"]]: - """Split leading post-join ``where_rows`` residuals by single node alias. - - #1729's connected-join lowering emits row predicates it cannot push into ``filter_dict`` - (e.g. ``toLower(i.interest) = toLower('fine dining')``) as leading ``where_rows`` ops in - ``post_join_chain``. The structural fast paths cannot apply a residual to the aggregated - frame, but a residual that references exactly ONE node alias the fast path materializes can - be applied to that alias's node set before counting. Returns ``(alias -> [expr, ...], - remaining_chain)`` when every leading residual is so attributable, or ``None`` when any - leading residual spans >1 alias, references a non-materialized alias, or is not a bare - ``expr`` residual -- the caller then declines to the slow path. - """ - from graphistry.compute.gfql.cypher.lowering import _expr_match_aliases - - ops = list(plan.post_join_chain.chain) - residuals: Dict[str, List[str]] = {} - consumed = 0 - for op in ops: - if not (isinstance(op, ASTCall) and op.function == "where_rows"): - break - params = op.params or {} - expr = params.get("expr") - if params.get("filter_dict") or not isinstance(expr, str): - return None - try: - aliases = _expr_match_aliases( - expr, alias_targets=alias_targets, params=None, field="where", line=0, column=0 - ) - except Exception: - return None - attributable = {alias for alias in aliases if alias in materialized_aliases} - if len(aliases) != 1 or len(attributable) != 1: - return None - residuals.setdefault(next(iter(attributable)), []).append(expr) - consumed += 1 - rest = Chain(ops[consumed:], where=plan.post_join_chain.where) - return residuals, rest - - -def _connected_join_apply_node_residuals( - base_graph: Plottable, - node_frame: DataFrameT, - alias: str, - exprs: Sequence[str], - node_col: str, - *, - engine: Engine, -) -> DataFrameT: - """Filter a fast-path node frame by single-alias post-join residual expressions. - - Reuses the row pipeline's ``where_rows`` evaluator (identical semantics to the slow path, - so toLower/etc. behave exactly as they would post-join) by aliasing the node columns to - ``alias.col`` and dispatching a where_rows chain, then renaming back. ``validate_schema`` is - disabled because the residual references flat ``alias.col`` columns rather than a bound - graph element. - """ - is_polars = "polars" in type(node_frame).__module__ - if is_polars: - aliased = node_frame.rename({col: f"{alias}.{col}" for col in node_frame.columns}) - else: - aliased = node_frame.rename(columns={col: f"{alias}.{col}" for col in node_frame.columns}) - from graphistry.compute.chain import chain as _chain_fn - - aliased_graph = base_graph.nodes(aliased, f"{alias}.{node_col}") - filtered_graph = _chain_fn( - aliased_graph, - [ASTCall("where_rows", {"expr": expr}) for expr in exprs], - engine=EngineAbstract(engine.value), - validate_schema=False, - ) - filtered = cast(DataFrameT, filtered_graph._nodes) - if is_polars: - return cast(DataFrameT, filtered.rename({f"{alias}.{col}": col for col in node_frame.columns})) - return cast(DataFrameT, filtered.rename(columns={f"{alias}.{col}": col for col in node_frame.columns})) - - -def _connected_join_filter_node_frames_by_residuals( - base_graph: Plottable, - residual_map: Mapping[str, Sequence[str]], - frames: Mapping[str, DataFrameT], - node_col: str, - *, - engine: Engine, -) -> Dict[str, DataFrameT]: - """Apply each alias's post-join residuals to its materialized node frame.""" - out: Dict[str, DataFrameT] = dict(frames) - for alias, exprs in residual_map.items(): - if alias in out and exprs: - out[alias] = _connected_join_apply_node_residuals( - base_graph, out[alias], alias, exprs, node_col, engine=engine - ) - return out - - -def _connected_join_two_star_fast_grouped_count( - base_graph: Plottable, - plan: ConnectedMatchJoinPlan, - *, - engine: Engine, - cache_store: Optional[Dict[str, Any]] = None, -) -> Optional[DataFrameT]: - # Per-execution cache scope: a fresh store when none is threaded in, so intra-query reuse - # is preserved without ever persisting caches on the caller's Plottable (BLOCKER 1). - if cache_store is None: - cache_store = {} - if len(plan.pattern_chains) != 2 or len(plan.pattern_shared_node_aliases) != 1: - return None - if len(plan.pattern_shared_node_aliases[0]) != 1: - return None - shared_alias = plan.pattern_shared_node_aliases[0][0] - - parsed = [] - for pattern_chain in plan.pattern_chains: - if pattern_chain.where: - return None - ops = list(pattern_chain.chain) - if len(ops) != 3: - return None - start_op, edge_op, end_op = ops - if not isinstance(start_op, ASTNode) or not isinstance(edge_op, ASTEdge) or not isinstance(end_op, ASTNode): - return None - if getattr(start_op, "query", None) is not None or getattr(end_op, "query", None) is not None: - return None - if not _is_connected_fast_single_hop(edge_op): - return None - start_alias = getattr(start_op, "_name", None) - end_alias = getattr(end_op, "_name", None) - if start_alias != shared_alias or not isinstance(end_alias, str): - return None - parsed.append((start_op, edge_op, end_op, end_alias)) - - first_start, first_edge, first_end, first_end_alias = parsed[0] - second_start, second_edge, second_end, second_end_alias = parsed[1] - if first_end_alias == second_end_alias: - return None - - alias_targets: Dict[str, ASTObject] = { - shared_alias: first_start, - first_end_alias: first_end, - second_end_alias: second_end, - } - materialized_aliases = {shared_alias, first_end_alias, second_end_alias} - split = _connected_join_two_star_split_residuals(plan, alias_targets, materialized_aliases) - if split is None: - return None - residual_map, rest_chain = split - - post_ops = [op for op in rest_chain.chain if isinstance(op, ASTCall)] - if len(post_ops) not in (2, 3, 4): - return None - if post_ops[0].function != "with_" or post_ops[1].function != "group_by": - return None - suffix = post_ops[2:] - order_call: Optional[ASTCall] = None - limit_call: Optional[ASTCall] = None - select_call: Optional[ASTCall] = None - for op in suffix: - if op.function == "order_by" and order_call is None and limit_call is None and select_call is None: - order_call = op - elif op.function == "limit" and limit_call is None and select_call is None: - limit_call = op - elif op.function == "select" and select_call is None: - select_call = op - else: - return None - - with_items_raw = post_ops[0].params.get("items") - group_keys_raw = post_ops[1].params.get("keys") - aggs_raw = post_ops[1].params.get("aggregations") - if not isinstance(with_items_raw, list) or not isinstance(group_keys_raw, list) or not isinstance(aggs_raw, list): - return None - if len(aggs_raw) != 1: - return None - agg = aggs_raw[0] - if not isinstance(agg, (tuple, list)) or len(agg) != 3 or str(agg[1]).lower() != "count": - return None - agg_alias = str(agg[0]) - agg_input = agg[2] - - with_items: Dict[str, Any] = {} - for item in with_items_raw: - if not isinstance(item, (tuple, list)) or len(item) != 2 or not isinstance(item[0], str): - return None - with_items[item[0]] = item[1] - # count(p) over the shared alias now lowers `p` to its identity column - # `p.__gfql_node_id__` (see _connected_join_alias_identity_expr); accept either the - # bare shared alias or that identity form -- the fast count is shared-node multiplicity - # regardless, so the identity rewrite does not change the computation. - shared_identity = f"{shared_alias}.{NODE_IDENTITY_COLUMN}" - if not isinstance(agg_input, str) or with_items.get(agg_input) not in (shared_alias, shared_identity): - return None - - group_keys = [str(key) for key in group_keys_raw] - group_prop_refs: List[Tuple[str, str]] = [] - output_group_keys: List[str] = [] - for key in group_keys: - expr = with_items.get(key) - if key == "__cypher_group__" and expr == 1: - continue - prop_ref = _property_ref(expr, (second_end_alias,)) - if prop_ref is None: - return None - output_group_keys.append(key) - group_prop_refs.append((key, prop_ref[1])) - - order_keys: List[Tuple[str, bool]] = [] - if order_call is not None: - raw_order = order_call.params.get("keys") - if not isinstance(raw_order, list): - return None - available = set(output_group_keys) | {agg_alias} - for item in raw_order: - if not isinstance(item, (tuple, list)) or len(item) != 2 or not isinstance(item[0], str): - return None - if item[0] not in available: - return None - direction = str(item[1]).lower() - if direction not in {"asc", "ascending", "desc", "descending"}: - return None - order_keys.append((item[0], direction in {"desc", "descending"})) - - limit_value: Optional[int] = None - if limit_call is not None: - raw_limit = limit_call.params.get("value") - if not isinstance(raw_limit, int) or raw_limit < 0: - return None - limit_value = raw_limit - - select_items: Optional[List[Tuple[str, str]]] = None - if select_call is not None: - raw_select = select_call.params.get("items") - if not isinstance(raw_select, list): - return None - select_items = [] - available = set(output_group_keys) | {agg_alias} - for item in raw_select: - if not isinstance(item, (tuple, list)) or len(item) != 2 or not isinstance(item[0], str) or not isinstance(item[1], str): - return None - if item[0] not in available: - return None - select_items.append((item[0], item[1])) - - nodes_obj = getattr(base_graph, "_nodes", None) - edges_obj = getattr(base_graph, "_edges", None) - node_col = getattr(base_graph, "_node", None) - src_col = getattr(base_graph, "_source", None) - dst_col = getattr(base_graph, "_destination", None) - if nodes_obj is None or edges_obj is None or node_col is None or src_col is None or dst_col is None: - return None - node_col = str(node_col) - src_col = str(src_col) - dst_col = str(dst_col) - if node_col not in nodes_obj.columns or src_col not in edges_obj.columns or dst_col not in edges_obj.columns: - return None - - nodes_source = cast(DataFrameT, nodes_obj) - nodes = df_to_engine(nodes_source, engine) - edges = cast(DataFrameT, edges_obj) - - if engine in POLARS_ENGINES: - import polars as pl - from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars - - # Residual node filters (e.g. toLower(...)) must be applied to the materialized node - # frames, so residual queries use the direct (non-cached) path -- the id/count caches - # re-derive node sets from filter_dict alone and would drop the residual. - if isinstance(nodes_obj, (pl.DataFrame, pl.LazyFrame)) and not residual_map: - shared_ids = _connected_join_cached_node_ids( - base_graph, - nodes_source, - [cast(Optional[dict], first_start.filter_dict), cast(Optional[dict], second_start.filter_dict)], - node_col=node_col, - engine=engine, - cache_store=cache_store, - ) - first_leaf_ids = _connected_join_cached_node_ids( - base_graph, - nodes_source, - [cast(Optional[dict], first_end.filter_dict)], - node_col=node_col, - engine=engine, - cache_store=cache_store, - ) - second_leaf_ids = _connected_join_cached_node_ids( - base_graph, - nodes_source, - [cast(Optional[dict], second_end.filter_dict)], - node_col=node_col, - engine=engine, - cache_store=cache_store, - ) - first_leaf_nodes = _connected_join_cached_node_filter(base_graph, nodes_source, cast(Optional[dict], first_end.filter_dict), engine=engine, cache_store=cache_store) - second_leaf_nodes = _connected_join_cached_node_filter(base_graph, nodes_source, cast(Optional[dict], second_end.filter_dict), engine=engine, cache_store=cache_store) - if shared_ids is None or first_leaf_ids is None or second_leaf_ids is None: - return None - else: - shared_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], first_start.filter_dict)) - shared_nodes = filter_by_dict_polars(shared_nodes, cast(Optional[dict], second_start.filter_dict)) - first_leaf_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], first_end.filter_dict)) - second_leaf_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], second_end.filter_dict)) - if residual_map: - _frames = _connected_join_filter_node_frames_by_residuals( - base_graph, - residual_map, - {shared_alias: shared_nodes, first_end_alias: first_leaf_nodes, second_end_alias: second_leaf_nodes}, - node_col, - engine=engine, - ) - shared_nodes, first_leaf_nodes, second_leaf_nodes = _frames[shared_alias], _frames[first_end_alias], _frames[second_end_alias] - shared_ids = shared_nodes.select(node_col).unique() - first_leaf_ids = first_leaf_nodes.select(node_col).unique() - second_leaf_ids = second_leaf_nodes.select(node_col).unique() - first_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], first_edge.edge_match), engine=engine, cache_store=cache_store) - second_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], second_edge.edge_match), engine=engine, cache_store=cache_store) - - for _, prop in group_prop_refs: - if prop not in second_leaf_nodes.columns: - return None - - def singleton_id(ids: Any) -> Any: - if not isinstance(ids, pl.DataFrame) or len(ids) != 1: - return _CACHE_MISSING - return ids.get_column(node_col)[0] - - first_leaf_singleton = singleton_id(first_leaf_ids) - second_leaf_singleton = singleton_id(second_leaf_ids) - - cached_left_counts: Optional[DataFrameT] = None - if first_leaf_singleton is not _CACHE_MISSING and not residual_map: - shared_ids_for_left = shared_ids.rename({node_col: shared_alias}) if node_col != shared_alias else shared_ids - cached_left_counts = _connected_join_cached_first_arm_shared_counts( - base_graph, - nodes_source, - edges, - first_edges, - shared_ids_for_left, - cast(Optional[dict], first_edge.edge_match), - cast(Optional[dict], first_start.filter_dict), - cast(Optional[dict], second_start.filter_dict), - first_leaf_singleton, - shared_alias=shared_alias, - src_col=src_col, - dst_col=dst_col, - engine=engine, - cache_store=cache_store, - ) - # Unreachable under the current lowering: _connected_join_cached_first_arm_shared_counts - # only returns None when a filter/edge/dst cache key is uncacheable, but every value - # that reaches this cached polars path is cacheable (scalar equality pushes as scalars; - # comparisons/toLower/ranges lower to residuals routed off the cached path). Kept as a - # defensive fallback; excluded from coverage since no legitimate query reaches it. - if cached_left_counts is None: # pragma: no cover - cached_left_counts = _connected_join_cached_singleton_dst_source_counts( - base_graph, - edges, - first_edges, - cast(Optional[dict], first_edge.edge_match), - first_leaf_singleton, - src_col=src_col, - dst_col=dst_col, - engine=engine, - cache_store=cache_store, - ) - if cached_left_counts is not None: - if shared_alias in cached_left_counts.columns: - left_counts = cached_left_counts - else: - left_counts = ( - cached_left_counts - .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") - .rename({src_col: shared_alias}) - ) - elif first_leaf_singleton is _CACHE_MISSING: - left_edges = ( - first_edges - .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") - .join(first_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") - ) - left_counts = ( - left_edges - .group_by(src_col) - .len("__left_count__") - .rename({src_col: shared_alias}) - ) - else: - left_edges = ( - first_edges - .filter(pl.col(dst_col) == first_leaf_singleton) - .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") - ) - left_counts = ( - left_edges - .group_by(src_col) - .len("__left_count__") - .rename({src_col: shared_alias}) - ) - cached_right_rows = None if residual_map else _connected_join_cached_second_arm_group_rows( - base_graph, - nodes_source, - edges, - second_edges, - shared_ids, - second_leaf_ids, - second_leaf_nodes, - cast(Optional[dict], first_start.filter_dict), - cast(Optional[dict], second_start.filter_dict), - cast(Optional[dict], second_end.filter_dict), - cast(Optional[dict], second_edge.edge_match), - second_leaf_singleton, - group_prop_refs, - output_group_keys, - node_col=node_col, - src_col=src_col, - dst_col=dst_col, - engine=engine, - cache_store=cache_store, - ) - if cached_right_rows is not None: - right_rows = cached_right_rows if src_col == shared_alias else cached_right_rows.rename({src_col: shared_alias}) - else: - if second_leaf_singleton is _CACHE_MISSING: - right_base = ( - second_edges - .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") - .join(second_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") - ) - else: - right_base = ( - second_edges - .filter(pl.col(dst_col) == second_leaf_singleton) - .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") - ) - if group_prop_refs: - lookup_key = "__gfql_fast_second_leaf_id__" - lookup_exprs = [pl.col(node_col).alias(lookup_key)] + [pl.col(prop).alias(out_col) for out_col, prop in group_prop_refs] - # Dedup by node identity to match the pandas drop_duplicates(subset=[node_col]); - # a duplicate node row would otherwise multiply the join and over-count. - second_lookup = second_leaf_nodes.select(lookup_exprs).unique(subset=[lookup_key]) - right_base = right_base.join(second_lookup, left_on=dst_col, right_on=lookup_key, how="inner") - right_rows = right_base.select([pl.col(src_col).alias(shared_alias)] + [pl.col(key) for key in output_group_keys]) - if not output_group_keys and len(left_counts) > 0 and bool(left_counts.select((pl.col("__left_count__") == 1).all()).item()): - matched_rows = right_rows.join(left_counts.select(shared_alias), on=shared_alias, how="semi") - out_df = matched_rows.select(pl.len().cast(pl.Int64).alias(agg_alias)) - else: - joined = right_rows.join(left_counts, on=shared_alias, how="inner") - if len(joined) == 0: - return cast(DataFrameT, joined.select([])) - if output_group_keys: - out_df = joined.group_by(output_group_keys, maintain_order=True).agg(pl.col("__left_count__").sum().cast(pl.Int64).alias(agg_alias)) - else: - out_df = joined.select(pl.col("__left_count__").sum().cast(pl.Int64).alias(agg_alias)) - if order_keys: - # openCypher orders NULL as the largest value: ASC -> nulls last, DESC -> nulls - # first. Polars defaults to nulls-first, which flips WHICH ROW an ORDER BY ... LIMIT - # returns, so pin nulls_last per key. - out_df = out_df.sort( - [key for key, _ in order_keys], - descending=[desc for _, desc in order_keys], - nulls_last=[not desc for _, desc in order_keys], - ) - if limit_value is not None: - out_df = out_df.head(limit_value) - if select_items is not None: - out_df = out_df.select([pl.col(src).alias(dst) for src, dst in select_items]) - return cast(DataFrameT, out_df) - - filter_engine = EngineAbstract(engine.value) - shared_nodes = filter_by_dict(nodes, cast(Optional[dict], first_start.filter_dict), engine=filter_engine) - shared_nodes = filter_by_dict(shared_nodes, cast(Optional[dict], second_start.filter_dict), engine=filter_engine) - first_leaf_nodes = filter_by_dict(nodes, cast(Optional[dict], first_end.filter_dict), engine=filter_engine) - second_leaf_nodes = filter_by_dict(nodes, cast(Optional[dict], second_end.filter_dict), engine=filter_engine) - if residual_map: - _frames = _connected_join_filter_node_frames_by_residuals( - base_graph, - residual_map, - {shared_alias: shared_nodes, first_end_alias: first_leaf_nodes, second_end_alias: second_leaf_nodes}, - node_col, - engine=engine, - ) - shared_nodes, first_leaf_nodes, second_leaf_nodes = _frames[shared_alias], _frames[first_end_alias], _frames[second_end_alias] - first_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], first_edge.edge_match), engine=engine, cache_store=cache_store) - second_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], second_edge.edge_match), engine=engine, cache_store=cache_store) - - shared_ids = shared_nodes[node_col].drop_duplicates() - first_leaf_ids = first_leaf_nodes[node_col].drop_duplicates() - second_leaf_ids = second_leaf_nodes[node_col].drop_duplicates() - for _, prop in group_prop_refs: - if prop not in second_leaf_nodes.columns: - return None - left_edges = first_edges[first_edges[src_col].isin(shared_ids) & first_edges[dst_col].isin(first_leaf_ids)] - left_counts = left_edges.groupby(src_col, sort=False).size().reset_index(name="__left_count__").rename(columns={src_col: shared_alias}) - right_base = second_edges[second_edges[src_col].isin(shared_ids) & second_edges[dst_col].isin(second_leaf_ids)] - if group_prop_refs: - lookup_key = "__gfql_fast_second_leaf_id__" - prop_cols = [] - for _, prop in group_prop_refs: - if prop not in prop_cols: - prop_cols.append(prop) - second_lookup = second_leaf_nodes[[node_col] + prop_cols].drop_duplicates(subset=[node_col]).rename(columns={node_col: lookup_key}) - for out_col, prop in group_prop_refs: - second_lookup[out_col] = second_lookup[prop] - right_base = right_base.merge(second_lookup[[lookup_key] + output_group_keys], left_on=dst_col, right_on=lookup_key, how="inner") - right_rows = right_base[[src_col] + output_group_keys].rename(columns={src_col: shared_alias}) - joined = right_rows.merge(left_counts, on=shared_alias, how="inner") - if len(joined) == 0: - return df_cons(engine)() - if output_group_keys: - out_df = joined.groupby(output_group_keys, sort=False, dropna=False)["__left_count__"].sum().reset_index(name=agg_alias) - else: - out_df = pd.DataFrame({agg_alias: [int(joined["__left_count__"].sum())]}) - if order_keys: - out_df = cast(DataFrameT, out_df.sort_values(by=[key for key, _ in order_keys], ascending=[not desc for _, desc in order_keys])) - if limit_value is not None: - out_df = cast(DataFrameT, out_df.head(limit_value)) - if select_items is not None: - out_df = out_df[[src for src, _ in select_items]].rename(columns={src: dst for src, dst in select_items}) - return df_to_engine(out_df.reset_index(drop=True), engine) - - -def _connected_join_two_star_fast_rows( - base_graph: Plottable, - plan: ConnectedMatchJoinPlan, - *, - engine: Engine, - cache_store: Optional[Dict[str, Any]] = None, -) -> Optional[DataFrameT]: - """Fast rows for T1 two-star connected joins. - - Supported shape: ``(p)-[r1]->(i), (p)-[r2]->(c)`` where both arms are fixed - forward one-hop patterns, share the same start node alias, have no residual - per-arm row filters, and downstream projections need at most properties from - the second leaf alias. The returned frame preserves row multiplicity, then - the normal row pipeline handles RETURN/GROUP/ORDER/LIMIT. - """ - # Per-execution cache scope (see _connected_join_two_star_fast_grouped_count); never - # persisted on the caller's Plottable. - if cache_store is None: - cache_store = {} - if len(plan.pattern_chains) != 2 or len(plan.pattern_shared_node_aliases) != 1: - return None - if len(plan.pattern_shared_node_aliases[0]) != 1: - return None - shared_alias = plan.pattern_shared_node_aliases[0][0] - - parsed = [] - for pattern_chain in plan.pattern_chains: - if pattern_chain.where: - return None - ops = list(pattern_chain.chain) - if len(ops) != 3: - return None - start_op, edge_op, end_op = ops - if not isinstance(start_op, ASTNode) or not isinstance(edge_op, ASTEdge) or not isinstance(end_op, ASTNode): - return None - if getattr(start_op, "query", None) is not None or getattr(end_op, "query", None) is not None: - return None - if not _is_connected_fast_single_hop(edge_op): - return None - start_alias = getattr(start_op, "_name", None) - end_alias = getattr(end_op, "_name", None) - if start_alias != shared_alias or not isinstance(end_alias, str): - return None - parsed.append((start_op, edge_op, end_op, end_alias)) - - first_start, first_edge, first_end, first_end_alias = parsed[0] - second_start, second_edge, second_end, second_end_alias = parsed[1] - if first_end_alias == second_end_alias: - return None - - attach_aliases = plan.pattern_attach_prop_aliases - if not attach_aliases or len(attach_aliases) != 2: - return None - normalized_attach_aliases: List[Tuple[str, ...]] = [] - for aliases in attach_aliases: - if aliases is None: - return None - normalized_attach_aliases.append(aliases) - needed_attach_aliases = {alias for aliases in normalized_attach_aliases for alias in aliases} - if any(alias != second_end_alias for alias in needed_attach_aliases): - return None - second_props_needed = _connected_join_post_property_columns(plan, second_end_alias) - # A bare aggregate over the second leaf (count(c) / count(DISTINCT c)) lowers `c` to its - # identity column c.__gfql_node_id__, which is NOT a materializable node property here -- - # fast_rows would drop it and post_join's count(c.__gfql_node_id__) would dereference a - # missing column. Decline to the (correct) slow path in that case. - if NODE_IDENTITY_COLUMN in second_props_needed: - return None - if second_end_alias in needed_attach_aliases and not second_props_needed: - return None - - nodes_obj = getattr(base_graph, "_nodes", None) - edges_obj = getattr(base_graph, "_edges", None) - node_col = getattr(base_graph, "_node", None) - src_col = getattr(base_graph, "_source", None) - dst_col = getattr(base_graph, "_destination", None) - if nodes_obj is None or edges_obj is None or node_col is None or src_col is None or dst_col is None: - return None - node_col = str(node_col) - src_col = str(src_col) - dst_col = str(dst_col) - if node_col not in nodes_obj.columns or src_col not in edges_obj.columns or dst_col not in edges_obj.columns: - return None - - nodes = df_to_engine(cast(DataFrameT, nodes_obj), engine) - edges = cast(DataFrameT, edges_obj) - - if engine in POLARS_ENGINES: - import polars as pl - from graphistry.compute.gfql.lazy.engine.polars.predicates import filter_by_dict_polars - - shared_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], first_start.filter_dict)) - shared_nodes = filter_by_dict_polars(shared_nodes, cast(Optional[dict], second_start.filter_dict)) - first_leaf_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], first_end.filter_dict)) - second_leaf_nodes = filter_by_dict_polars(nodes, cast(Optional[dict], second_end.filter_dict)) - first_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], first_edge.edge_match), engine=engine, cache_store=cache_store) - second_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], second_edge.edge_match), engine=engine, cache_store=cache_store) - - shared_ids = shared_nodes.select(node_col).unique() - first_leaf_ids = first_leaf_nodes.select(node_col).unique() - second_leaf_ids = second_leaf_nodes.select(node_col).unique() - second_props = [col for col in second_props_needed if col in second_leaf_nodes.columns] - - left_rows = ( - first_edges - .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") - .join(first_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") - .select(pl.col(src_col).alias(shared_alias)) - ) - right_base = ( - second_edges - .join(shared_ids, left_on=src_col, right_on=node_col, how="semi") - .join(second_leaf_ids, left_on=dst_col, right_on=node_col, how="semi") - ) - if second_props: - second_lookup_key = "__gfql_fast_second_leaf_id__" - # Dedup by node identity (matches pandas drop_duplicates(subset=[node_col])); a - # duplicate node row would otherwise multiply the join and inflate row multiplicity. - second_lookup = second_leaf_nodes.select([node_col] + second_props).unique(subset=[node_col]).rename({node_col: second_lookup_key}) - second_lookup = second_lookup.rename({col: f"{second_end_alias}.{col}" for col in second_props}) - right_base = right_base.join(second_lookup, left_on=dst_col, right_on=second_lookup_key, how="inner") - right_select = [pl.col(src_col).alias(shared_alias)] + [pl.col(f"{second_end_alias}.{col}") for col in second_props] - right_rows = right_base.select(right_select) - return cast(DataFrameT, left_rows.join(right_rows, on=shared_alias, how="inner")) - - filter_engine = EngineAbstract(engine.value) - shared_nodes = filter_by_dict(nodes, cast(Optional[dict], first_start.filter_dict), engine=filter_engine) - shared_nodes = filter_by_dict(shared_nodes, cast(Optional[dict], second_start.filter_dict), engine=filter_engine) - first_leaf_nodes = filter_by_dict(nodes, cast(Optional[dict], first_end.filter_dict), engine=filter_engine) - second_leaf_nodes = filter_by_dict(nodes, cast(Optional[dict], second_end.filter_dict), engine=filter_engine) - first_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], first_edge.edge_match), engine=engine, cache_store=cache_store) - second_edges = _connected_join_cached_edge_filter(base_graph, edges, cast(Optional[dict], second_edge.edge_match), engine=engine, cache_store=cache_store) - - shared_ids = shared_nodes[node_col].drop_duplicates() - first_leaf_ids = first_leaf_nodes[node_col].drop_duplicates() - second_leaf_ids = second_leaf_nodes[node_col].drop_duplicates() - second_props = [col for col in second_props_needed if col in second_leaf_nodes.columns] - - left_edges = first_edges[first_edges[src_col].isin(shared_ids) & first_edges[dst_col].isin(first_leaf_ids)] - left_rows = left_edges[[src_col]].rename(columns={src_col: shared_alias}) - right_base = second_edges[second_edges[src_col].isin(shared_ids) & second_edges[dst_col].isin(second_leaf_ids)] - if second_props: - second_lookup_key = "__gfql_fast_second_leaf_id__" - second_lookup_cols = [node_col] + second_props - second_lookup = second_leaf_nodes[second_lookup_cols].drop_duplicates(subset=[node_col]).rename( - columns={ - node_col: second_lookup_key, - **{col: f"{second_end_alias}.{col}" for col in second_props}, - } - ) - right_base = right_base.merge(second_lookup, left_on=dst_col, right_on=second_lookup_key, how="inner") - right_cols = [src_col] + [f"{second_end_alias}.{col}" for col in second_props] - right_rows = right_base[right_cols].rename(columns={src_col: shared_alias}) - return cast(DataFrameT, left_rows.merge(right_rows, on=shared_alias, how="inner")) +# Fast-path specialization helpers extracted to gfql_fast_paths.py (pure move; #1731). +from .gfql_fast_paths import ( + _connected_join_two_star_fast_grouped_count, + _connected_join_two_star_fast_rows, + _execute_single_hop_grouped_aggregate_fast_path, + _execute_two_hop_count_fast_path, +) def _apply_connected_match_join( base_graph: Plottable, @@ -1791,7 +692,6 @@ def _apply_graph_residual_filters( graph = graph.edges(cast(DataFrameT, edges_df.loc[edge_mask])) return graph - def _execute_graph_constructor_compiled( base_graph: Plottable, chain: Chain, @@ -2038,6 +938,12 @@ def _execute_compiled_query_via_physical_plan( ) if physical_plan.route in ("same_path", "row_pipeline"): + fast_grouped = _execute_single_hop_grouped_aggregate_fast_path(base_graph, compiled_query.chain, engine=engine) + if fast_grouped is not None: + return fast_grouped + fast_count = _execute_two_hop_count_fast_path(base_graph, compiled_query.chain, engine=engine) + if fast_count is not None: + return fast_count return _execute_compiled_query_chain_non_union( base_graph, compiled_query=compiled_query, @@ -2504,24 +1410,124 @@ def detect_query_type(query: Any) -> QueryType: return "single" +_COMPILED_STRING_QUERY_CACHE_ATTR = "_gfql_compiled_string_query_cache" +_COMPILED_STRING_QUERY_CACHE_MAX = 128 + + +def _compile_cache_value_key(value: Any) -> Optional[Any]: + if value is None: + return ("none",) + if isinstance(value, bool): + return ("bool", value) + if isinstance(value, int): + return ("int", value) + if isinstance(value, float): + return ("float", value) + if isinstance(value, str): + return ("str", value) + if isinstance(value, (list, tuple)): + items = [] + for item in value: + item_key = _compile_cache_value_key(item) + if item_key is None: + return None + items.append(item_key) + return ("list", tuple(items)) + if isinstance(value, Mapping): + items = [] + seen_keys = set() + for key, item in value.items(): + key_str = str(key) + if key_str in seen_keys: + return None + seen_keys.add(key_str) + item_key = _compile_cache_value_key(item) + if item_key is None: + return None + items.append((key_str, item_key)) + return ("mapping", tuple(sorted(items))) + return None + + +def _compile_cache_params_key(params: Optional[Mapping[str, Any]]) -> Optional[Tuple[Tuple[str, Any], ...]]: + if not params: + return () + items = [] + seen_keys = set() + for key, value in params.items(): + key_str = str(key) + if key_str in seen_keys: + return None + seen_keys.add(key_str) + value_key = _compile_cache_value_key(value) + if value_key is None: + return None + items.append((key_str, value_key)) + return tuple(sorted(items)) + + +def _node_dtypes_cache_key( + node_dtypes: Optional[NodeDtypes], +) -> Optional[Tuple[Tuple[str, str], ...]]: + """Hashable key for node_dtypes so the compile cache never reuses an engine-specific + pushdown plan across differing dtype views (e.g. pandas numpy dtypes vs polars dtypes).""" + if node_dtypes is None: + return None + return tuple(sorted((str(col), str(dtype)) for col, dtype in node_dtypes.items())) + + +def _compile_string_query_cache( + cache_owner: Optional[Plottable], +) -> Optional[Dict[Tuple[str, str, Tuple[Tuple[str, Any], ...], Optional[Tuple[Tuple[str, str], ...]]], Any]]: + if cache_owner is None: + return None + try: + cache = getattr(cache_owner, _COMPILED_STRING_QUERY_CACHE_ATTR, None) + if cache is None: + cache = {} + setattr(cache_owner, _COMPILED_STRING_QUERY_CACHE_ATTR, cache) + if isinstance(cache, dict): + return cache + except Exception: + return None + return None + + def _compile_string_query( query: str, *, language: Optional[Literal["cypher", "gremlin"]], params: Optional[Mapping[str, Any]], + cache_owner: Optional[Plottable] = None, node_dtypes: Optional[NodeDtypes] = None, ) -> Any: query_language = language or "cypher" if query_language != "cypher": raise GFQLValidationError( ErrorCode.E108, - f"Unsupported GFQL string language '{query_language}'", + f"Unsupported GFQL string language {query_language!r}", field="language", value=query_language, - suggestion="Use language='cypher' for now; Gremlin string compilation is not implemented yet.", + suggestion="Use language=\"cypher\" for now; Gremlin string compilation is not implemented yet.", language="gfql", ) - return compile_cypher_query(parse_cypher(query), params=params, node_dtypes=node_dtypes) + params_key = _compile_cache_params_key(params) + # node_dtypes (from #1730/#1729 pushdown) makes compilation engine-dependent: the same + # (query, params) yields a different pushdown plan under pandas vs polars dtypes. The + # compile cache (#1731) must therefore key on node_dtypes too, or a plan compiled for one + # engine would be wrongly reused for another on the same graph. + node_dtypes_key = _node_dtypes_cache_key(node_dtypes) + cache = _compile_string_query_cache(cache_owner) if params_key is not None else None + cache_key = (query_language, query, params_key, node_dtypes_key) if params_key is not None else None + if cache is not None and cache_key is not None and cache_key in cache: + return cache[cache_key] + + compiled = compile_cypher_query(parse_cypher(query), params=params, node_dtypes=node_dtypes) + if cache is not None and cache_key is not None: + if cache_key not in cache and len(cache) >= _COMPILED_STRING_QUERY_CACHE_MAX: + cache.clear() + cache[cache_key] = compiled + return compiled def _compile_value_repr(value: Any) -> str: @@ -2821,6 +1827,7 @@ def gfql(self: Plottable, query, language=language, params=params, + cache_owner=dispatch_self, node_dtypes=_node_dtypes_for_pushdown(self, engine), ) except GFQLValidationError as exc: diff --git a/graphistry/tests/compute/gfql/cypher/test_compiler_policy.py b/graphistry/tests/compute/gfql/cypher/test_compiler_policy.py index f2e470007f..88f7d62845 100644 --- a/graphistry/tests/compute/gfql/cypher/test_compiler_policy.py +++ b/graphistry/tests/compute/gfql/cypher/test_compiler_policy.py @@ -1,12 +1,13 @@ from __future__ import annotations -from typing import List, cast +from typing import Any, List, cast import pandas as pd import pytest from graphistry.compute.exceptions import ErrorCode, GFQLValidationError from graphistry.compute.gfql.policy import CompileSummary, PolicyContext, PolicyException +from graphistry.compute import gfql_unified as gfql_unified_module from graphistry.tests.test_compute import CGFull @@ -194,3 +195,75 @@ def deny(ctx: PolicyContext) -> None: assert calls == ["precompile"] assert exc_info.value.phase == "precompile" assert exc_info.value.query_type == "chain" + + +def test_compiled_string_query_cache_reuses_identical_query_per_graph(monkeypatch: pytest.MonkeyPatch) -> None: + # _compile_string_query now parses+compiles via parse_cypher/compile_cypher_query (a compile + # runs parse_cypher exactly once per cache MISS), so count compiles by spying parse_cypher. + compile_calls: List[str] = [] + original_parse = gfql_unified_module.parse_cypher + + def spy_parse(query: str, *args: Any, **kwargs: Any) -> Any: + compile_calls.append(query) + return original_parse(query, *args, **kwargs) + + monkeypatch.setattr(gfql_unified_module, "parse_cypher", spy_parse) + graph = _mk_graph() + query = "UNWIND [1, 2, 3] AS x RETURN x ORDER BY x" + + first = graph.gfql(query) + second = graph.gfql(query) + _mk_graph().gfql(query) + + assert first._nodes[["x"]].to_dict(orient="records") == [{"x": 1}, {"x": 2}, {"x": 3}] + assert second._nodes[["x"]].to_dict(orient="records") == [{"x": 1}, {"x": 2}, {"x": 3}] + # graph1 first call compiles; graph1 second call is a cache hit; the fresh graph compiles again. + assert compile_calls == [query, query] + + +def test_compiled_string_query_cache_keys_include_params(monkeypatch: pytest.MonkeyPatch) -> None: + compile_params: List[Any] = [] + original_compile = gfql_unified_module.compile_cypher_query + + def spy_compile(parsed: Any, *args: Any, **kwargs: Any) -> Any: + compile_params.append(kwargs.get("params")) + return original_compile(parsed, *args, **kwargs) + + monkeypatch.setattr(gfql_unified_module, "compile_cypher_query", spy_compile) + graph = _mk_graph() + query = "RETURN $value AS value" + + one = graph.gfql(query, params={"value": 1}) + two = graph.gfql(query, params={"value": 2}) + one_again = graph.gfql(query, params={"value": 1}) + + assert one._nodes.to_dict(orient="records") == [{"value": 1}] + assert two._nodes.to_dict(orient="records") == [{"value": 2}] + assert one_again._nodes.to_dict(orient="records") == [{"value": 1}] + assert compile_params == [{"value": 1}, {"value": 2}] + + +def test_compiled_string_query_cache_hit_still_fires_compile_policies(monkeypatch: pytest.MonkeyPatch) -> None: + compile_calls: List[str] = [] + policy_calls: List[str] = [] + original_parse = gfql_unified_module.parse_cypher + + def spy_compile(query: str, *args: Any, **kwargs: Any) -> Any: + compile_calls.append(query) + return original_parse(query, *args, **kwargs) + + def observe(ctx: PolicyContext) -> None: + policy_calls.append(cast(str, ctx["phase"])) + if ctx["phase"] == "postcompile": + assert ctx["success"] is True + assert isinstance(ctx["compile"], CompileSummary) + + monkeypatch.setattr(gfql_unified_module, "parse_cypher", spy_compile) + graph = _mk_graph() + query = "RETURN 1 AS value" + + graph.gfql(query, policy={"precompile": observe, "postcompile": observe}) + graph.gfql(query, policy={"precompile": observe, "postcompile": observe}) + + assert compile_calls == [query] + assert policy_calls == ["precompile", "postcompile", "precompile", "postcompile"] diff --git a/graphistry/tests/compute/gfql/cypher/test_lowering.py b/graphistry/tests/compute/gfql/cypher/test_lowering.py index d060710516..093c727fcc 100644 --- a/graphistry/tests/compute/gfql/cypher/test_lowering.py +++ b/graphistry/tests/compute/gfql/cypher/test_lowering.py @@ -40,15 +40,35 @@ networkx_normalized_value_columns, ) from graphistry.plugins.networkx.policy import NETWORKX_SCIPY_EXTRA_REQUIREMENTS, NETWORKX_VERSION_SPEC, SCIPY_VERSION_SPEC -from graphistry.compute.gfql.cypher.ast import ExpressionText, OrderByClause, OrderItem, ReturnClause, ReturnItem, SourceSpan +from graphistry.compute.gfql.cypher.ast import ExpressionText, LabelRef, OrderByClause, OrderItem, ParameterRef, PropertyRef, ReturnClause, ReturnItem, SourceSpan, WherePredicate +import graphistry.compute.gfql.cypher.lowering as lowering_module from graphistry.compute.gfql.cypher.lowering import CompiledCypherExecutionExtras, CompiledCypherGraphQuery, compile_cypher_query -from graphistry.compute.gfql.cypher.lowering import ConnectedMatchJoinPlan, _logical_plan_route_for_query -from graphistry.compute.gfql_unified import ( +from graphistry.compute.gfql.cypher.lowering import ( + ConnectedMatchJoinPlan, + _AggregateSpec, + _active_match_alias_for_stage, + _aggregate_runtime_spec, + _binding_prop_alias_set, + _distinct_aggregate_expr_text, + _clause_has_mixed_aggregate_item, + _expr_property_access_node_aliases, + _is_multi_source_match_alias_boundary_error, + _logical_plan_route_for_query, + _projection_ref_from_expr_safe, + _render_row_where_operand_text, + _row_where_predicate_text, + _whole_row_group_entity_expr, + _whole_row_group_key_expr, +) +from graphistry.compute.gfql_fast_paths import ( _connected_join_cached_edge_filter, _connected_join_cached_singleton_dst_source_counts, _connected_join_post_property_columns, _connected_join_two_star_fast_grouped_count, _connected_join_two_star_fast_rows, + _execute_single_hop_grouped_aggregate_fast_path, + _execute_two_hop_count_fast_path, + _filter_nodes_for_fast_count, _connected_join_two_star_split_residuals, ) from graphistry.compute.gfql.frontends.cypher.binder import FrontendBinder @@ -14947,6 +14967,225 @@ def test_issue_1273_multi_source_grouped_aggregate(agg: str, expected: list) -> assert result._nodes.to_dict(orient="records") == expected +@pytest.mark.parametrize("engine", ["pandas", "polars"]) +def test_t4_single_hop_grouped_fast_path_q1_count_bound_alias(engine: str) -> None: + if engine == "polars": + pytest.importorskip("polars") + nodes = pd.DataFrame({ + "id": [0, 1, 2], + "node_type": ["Person", "Person", "Person"], + }) + edges = pd.DataFrame({ + "s": [0, 1, 0], + "d": [2, 2, 1], + "rel": ["FOLLOWS", "FOLLOWS", "FOLLOWS"], + }) + query = ( + "MATCH (f {node_type:'Person'})-[{rel:'FOLLOWS'}]->(p {node_type:'Person'}) " + "RETURN p.id AS personID, count(f) AS numFollowers " + "ORDER BY numFollowers DESC, personID LIMIT 3" + ) + + graph = _mk_graph(nodes, edges) + compiled = cast(CompiledCypherQuery, compile_cypher(query)) + fast = _execute_single_hop_grouped_aggregate_fast_path(graph, compiled.chain, engine=engine) + assert fast is not None + + result = graph.gfql(query, engine=engine) + assert _to_pandas_df(result._nodes).to_dict(orient="records") == [ + {"personID": 2, "numFollowers": 2}, + {"personID": 1, "numFollowers": 1}, + ] + + +@pytest.mark.parametrize("engine", ["pandas", "polars"]) +def test_single_hop_grouped_fast_path_orders_nulls_as_largest(engine: str) -> None: + # The single-hop grouped-aggregate fast path's polars sort omitted nulls_last; polars + # defaults nulls-first while openCypher orders NULL as the largest value. So ORDER BY the + # group key ASC ... LIMIT 1 must return the smallest NON-null group (SF), not the NULL group. + if engine == "polars": + pytest.importorskip("polars") + nodes = pd.DataFrame({ + "id": ["p0", "p1", "p2", "c0", "c1", "c2"], + "node_type": ["Person", "Person", "Person", "City", "City", "City"], + "city": [None, None, None, "SF", "NY", None], # c2 has a NULL city + }) + edges = pd.DataFrame({ + "s": ["p0", "p1", "p2"], + "d": ["c0", "c1", "c2"], + "rel": ["LIVES_IN", "LIVES_IN", "LIVES_IN"], + }) + graph = _mk_graph(nodes, edges) + query = ( + "MATCH (p {node_type:'Person'})-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) " + "RETURN count(p) AS n, c.city AS city ORDER BY city ASC LIMIT 1" + ) + compiled = cast(CompiledCypherQuery, compile_cypher(query)) + assert _execute_single_hop_grouped_aggregate_fast_path(graph, compiled.chain, engine=Engine(engine)) is not None + assert _to_pandas_df(graph.gfql(query, engine=engine)._nodes).to_dict(orient="records") == [{"n": 1, "city": "NY"}] + + +@pytest.mark.parametrize("engine", ["pandas", "polars"]) +def test_single_hop_grouped_fast_path_orders_nulls_first_on_desc(engine: str) -> None: + # Amplification (round 001): the ASC sibling above only exercised ASC, where pandas' + # default na_position='last' is coincidentally correct. On DESC, openCypher orders NULL as + # the LARGEST value, so the NULL-city group must sort FIRST. The pandas fast-path branch used + # a single sort_values(ascending=[...]) with a SCALAR na_position (default 'last'), so it put + # the NULL group last on DESC -- a silent pandas/polars divergence (the polars branch pins + # nulls_last per key) that returns the WRONG ORDER BY ... DESC LIMIT top-k. Hand-derived + # openCypher order (NULL largest, DESC): NULL, then SF (count 2), then NY (count 1). + if engine == "polars": + pytest.importorskip("polars") + nodes = pd.DataFrame({ + "id": ["p0", "p1", "p2", "p3", "c0", "c1", "c2"], + "node_type": ["Person", "Person", "Person", "Person", "City", "City", "City"], + "city": [None, None, None, None, "SF", "NY", None], # c2 has a NULL city + }) + edges = pd.DataFrame({ + "s": ["p0", "p3", "p1", "p2"], + "d": ["c0", "c0", "c1", "c2"], # SF<-{p0,p3}=2, NY<-{p1}=1, NULL<-{p2}=1 + "rel": ["LIVES_IN", "LIVES_IN", "LIVES_IN", "LIVES_IN"], + }) + graph = _mk_graph(nodes, edges) + base = ( + "MATCH (p {node_type:'Person'})-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) " + "RETURN count(p) AS n, c.city AS city ORDER BY city DESC" + ) + compiled = cast(CompiledCypherQuery, compile_cypher(base)) + assert _execute_single_hop_grouped_aggregate_fast_path(graph, compiled.chain, engine=Engine(engine)) is not None + + def _recs(nodes: Any) -> list: + # Normalize the null-city representation (pandas nan vs polars None) so the ordered + # comparison tests ORDER, not null spelling. + df = _to_pandas_df(nodes) + return [{k: (None if pd.isna(v) else v) for k, v in rec.items()} for rec in df.to_dict(orient="records")] + + # Full DESC order: NULL group first (largest), then SF, then NY. + assert _recs(graph.gfql(base, engine=engine)._nodes) == [ + {"n": 1, "city": None}, + {"n": 2, "city": "SF"}, + {"n": 1, "city": "NY"}, + ] + # DESC LIMIT 1 must return the NULL group (openCypher NULL == largest), not SF. + limited = base + " LIMIT 1" + assert _recs(graph.gfql(limited, engine=engine)._nodes) == [{"n": 1, "city": None}] + + +@pytest.mark.parametrize("engine", ["pandas", "polars"]) +def test_t3_single_hop_grouped_fast_path_q3_avg_source_prop_by_dest_prop(engine: str) -> None: + if engine == "polars": + pytest.importorskip("polars") + nodes = pd.DataFrame({ + "id": [0, 1, 2, 10, 11], + "node_type": ["Person", "Person", "Person", "City", "City"], + "age": [20, 30, 40, None, None], + "city": [None, None, None, "NYC", "LA"], + "country": [None, None, None, "United States", "United States"], + }) + edges = pd.DataFrame({ + "s": [0, 1, 2], + "d": [10, 10, 11], + "rel": ["LIVES_IN", "LIVES_IN", "LIVES_IN"], + }) + query = ( + "MATCH (p {node_type:'Person'})-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) " + "WHERE c.country = 'United States' " + "RETURN c.city AS city, avg(p.age) AS averageAge " + "ORDER BY averageAge, city LIMIT 5" + ) + + graph = _mk_graph(nodes, edges) + compiled = cast(CompiledCypherQuery, compile_cypher(query)) + fast = _execute_single_hop_grouped_aggregate_fast_path(graph, compiled.chain, engine=engine) + assert fast is not None + + result = graph.gfql(query, engine=engine) + assert _to_pandas_df(result._nodes).to_dict(orient="records") == [ + {"city": "NYC", "averageAge": 25.0}, + {"city": "LA", "averageAge": 40.0}, + ] + + +@pytest.mark.parametrize("engine", ["pandas", "polars"]) +def test_t3_single_hop_grouped_fast_path_q4_count_preserves_edge_multiplicity(engine: str) -> None: + if engine == "polars": + pytest.importorskip("polars") + nodes = pd.DataFrame({ + "id": [0, 1, 2, 3, 10, 11], + "node_type": ["Person", "Person", "Person", "Person", "City", "City"], + "age": [25, 30, 35, 41, None, None], + "country": [None, None, None, None, "United States", "United Kingdom"], + }) + edges = pd.DataFrame({ + "s": [0, 1, 2, 2, 3], + "d": [10, 10, 11, 11, 10], + "rel": ["LIVES_IN", "LIVES_IN", "LIVES_IN", "LIVES_IN", "LIVES_IN"], + }) + query = ( + "MATCH (p {node_type:'Person'})-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) " + "WHERE p.age >= 30 AND p.age <= 40 " + "RETURN c.country AS countries, count(*) AS personCounts " + "ORDER BY personCounts DESC, countries LIMIT 3" + ) + + graph = _mk_graph(nodes, edges) + compiled = cast(CompiledCypherQuery, compile_cypher(query)) + fast = _execute_single_hop_grouped_aggregate_fast_path(graph, compiled.chain, engine=engine) + assert fast is not None + + result = graph.gfql(query, engine=engine) + assert _to_pandas_df(result._nodes).to_dict(orient="records") == [ + {"countries": "United Kingdom", "personCounts": 2}, + {"countries": "United States", "personCounts": 1}, + ] + + +def test_t3_single_hop_grouped_fast_path_pandas_numeric_aggregates() -> None: + nodes = pd.DataFrame({ + "id": [0, 1, 2, 10, 11], + "node_type": ["Person", "Person", "Person", "City", "City"], + "age": [20, 30, 40, None, None], + "city": [None, None, None, "NYC", "LA"], + }) + edges = pd.DataFrame({ + "s": [0, 1, 2], + "d": [10, 10, 11], + "rel": ["LIVES_IN", "LIVES_IN", "LIVES_IN"], + }) + query = ( + "MATCH (p {node_type:'Person'})-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) " + "RETURN c.city AS city, count(p.age) AS ageCount, sum(p.age) AS ageSum, " + "min(p.age) AS minAge, max(p.age) AS maxAge ORDER BY city" + ) + + graph = _mk_graph(nodes, edges) + compiled = cast(CompiledCypherQuery, compile_cypher(query)) + fast = _execute_single_hop_grouped_aggregate_fast_path(graph, compiled.chain, engine="pandas") + assert fast is not None + assert _to_pandas_df(fast._nodes).to_dict(orient="records") == [ + {"city": "LA", "ageCount": 1, "ageSum": 40, "minAge": 40, "maxAge": 40}, + {"city": "NYC", "ageCount": 2, "ageSum": 50, "minAge": 20, "maxAge": 30}, + ] + + +def test_t3_single_hop_grouped_fast_path_pandas_missing_property_declines() -> None: + nodes = pd.DataFrame({ + "id": [0, 10], + "node_type": ["Person", "City"], + "age": [20, None], + "city": [None, "NYC"], + }) + edges = pd.DataFrame({"s": [0], "d": [10], "rel": ["LIVES_IN"]}) + query = ( + "MATCH (p {node_type:'Person'})-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) " + "RETURN c.missing AS missing, count(*) AS n" + ) + + graph = _mk_graph(nodes, edges) + compiled = cast(CompiledCypherQuery, compile_cypher(query)) + assert _execute_single_hop_grouped_aggregate_fast_path(graph, compiled.chain, engine="pandas") is None + + def test_issue_1712_connected_comma_pattern_where_intersects() -> None: """#1712: a connected comma-pattern sharing a node alias with a WHERE on a leaf alias must intersect both patterns (the WHERE was silently dropped on the @@ -16740,6 +16979,107 @@ def test_t1_connected_comma_edge_filter_cache_reuses_simple_edge_match() -> None assert not [attr for attr in vars(graph) if attr.startswith("_gfql_connected_join")] +@pytest.mark.parametrize("engine", ["pandas", "polars"]) +def test_t2_two_hop_count_fast_path_q8_shape_preserves_multiplicity(engine: str) -> None: + if engine == "polars": + pytest.importorskip("polars") + nodes = pd.DataFrame({ + "id": [0, 1, 2, 3], + "node_type": ["Person", "Person", "Person", "Person"], + "age": [20, 30, 40, 50], + }) + edges = pd.DataFrame({ + "s": [0, 0, 1, 1, 1, 2], + "d": [1, 1, 2, 2, 3, 3], + "rel": ["FOLLOWS", "FOLLOWS", "FOLLOWS", "FOLLOWS", "FOLLOWS", "FOLLOWS"], + }) + query = ( + "MATCH (a {node_type:'Person'})-[{rel:'FOLLOWS'}]->(b {node_type:'Person'})" + "-[{rel:'FOLLOWS'}]->(d {node_type:'Person'}) " + "RETURN count(*) AS numPaths" + ) + + result = _mk_graph(nodes, edges).gfql(query, engine=engine) + assert _to_pandas_df(result._nodes).to_dict(orient="records") == [{"numPaths": 8}] + + +@pytest.mark.parametrize("engine", ["pandas", "polars"]) +def test_t2_two_hop_count_fast_path_q9_shape_filters_middle_and_end(engine: str) -> None: + if engine == "polars": + pytest.importorskip("polars") + nodes = pd.DataFrame({ + "id": [0, 1, 2, 3, 4], + "node_type": ["Person", "Person", "Person", "Person", "Person"], + "age": [20, 30, 60, 40, 10], + }) + edges = pd.DataFrame({ + "s": [0, 4, 1, 1, 2, 2], + "d": [1, 1, 2, 3, 3, 4], + "rel": ["FOLLOWS", "FOLLOWS", "FOLLOWS", "FOLLOWS", "FOLLOWS", "FOLLOWS"], + }) + query = ( + "MATCH (a {node_type:'Person'})-[{rel:'FOLLOWS'}]->(b {node_type:'Person'})" + "-[{rel:'FOLLOWS'}]->(d {node_type:'Person'}) " + "WHERE b.age < 50 AND d.age > 25 RETURN count(*) AS numPaths" + ) + + result = _mk_graph(nodes, edges).gfql(query, engine=engine) + assert _to_pandas_df(result._nodes).to_dict(orient="records") == [{"numPaths": 5}] + + +def test_t2_two_hop_count_fast_path_empty_count_returns_zero() -> None: + nodes = pd.DataFrame({"id": [0, 1], "node_type": ["Person", "Person"]}) + edges = pd.DataFrame({"s": [0], "d": [1], "rel": ["FOLLOWS"]}) + query = ( + "MATCH (a {node_type:'Person'})-[{rel:'FOLLOWS'}]->(b {node_type:'Person'})" + "-[{rel:'FOLLOWS'}]->(d {node_type:'Person'}) " + "RETURN count(*) AS numPaths" + ) + + compiled = cast(CompiledCypherQuery, compile_cypher(query)) + result = _execute_two_hop_count_fast_path(_mk_graph(nodes, edges), compiled.chain, engine="pandas") + + assert result is not None + assert result._nodes.to_dict(orient="records") == [{"numPaths": 0}] + + +@pytest.mark.parametrize("engine", ["pandas", "polars"]) +def test_t5_two_hop_equal_domain_degree_cache_reuses_counts(engine: str) -> None: + if engine == "polars": + pytest.importorskip("polars") + nodes = pd.DataFrame({ + "id": [0, 1, 2, 3], + "node_type": ["Person", "Person", "Person", "City"], + }) + edges = pd.DataFrame({ + "s": [0, 0, 1, 1, 2, 2, 3], + "d": [1, 1, 2, 3, 0, 3, 0], + "rel": ["FOLLOWS", "FOLLOWS", "FOLLOWS", "FOLLOWS", "FOLLOWS", "LIKES", "FOLLOWS"], + }) + query = ( + "MATCH (a {node_type:'Person'})-[{rel:'FOLLOWS'}]->(b {node_type:'Person'})" + "-[{rel:'FOLLOWS'}]->(d {node_type:'Person'}) " + "RETURN count(*) AS numPaths" + ) + graph = _mk_graph(nodes, edges) + compiled = cast(CompiledCypherQuery, compile_cypher(query)) + + first = _execute_two_hop_count_fast_path(graph, compiled.chain, engine=engine) + assert first is not None + assert _to_pandas_df(first._nodes).to_dict(orient="records") == [{"numPaths": 5}] + + cache = getattr(graph, "_gfql_two_hop_equal_domain_degree_counts_cache") + assert isinstance(cache, dict) + assert len(cache) == 1 + cached_counts = next(iter(cache.values())) + + second = _execute_two_hop_count_fast_path(graph, compiled.chain, engine=engine) + assert second is not None + assert _to_pandas_df(second._nodes).to_dict(orient="records") == [{"numPaths": 5}] + assert len(cache) == 1 + assert next(iter(cache.values())) is cached_counts + + def test_issue_1413_ic3_entity_membership_positive_same_city_friend_only() -> None: graph = _mk_ic3_cross_country_shape_graph() result = graph.gfql( @@ -19324,6 +19664,69 @@ def test_count_table_frame_op_error_and_empty_paths() -> None: assert out3._nodes.to_dict(orient="records") == [{"c": 0}] +def test_connected_join_private_coverage_helpers_pandas() -> None: + edges = pd.DataFrame( + { + "s": ["a", "a", "b", "c"], + "d": ["z", "z", "z", "z"], + "kind": ["x", "x", "x", "y"], + } + ) + graph = _mk_graph(pd.DataFrame({"id": ["a", "b", "c", "z"]}), edges) + x_edges = edges[edges["kind"] == "x"] + + # Cache reuse is now scoped to a per-execution cache_store (never the Plottable, see + # BLOCKER 1 / #1730): a shared store returns the identical cached frame; without it, calls + # recompute. + cache_store: dict = {} + counts = _connected_join_cached_singleton_dst_source_counts( + graph, + graph._edges, + x_edges, + {"kind": "x"}, + "z", + src_col="s", + dst_col="d", + engine=Engine.PANDAS, + cache_store=cache_store, + ) + assert counts is not None + assert counts.sort_values("s").to_dict(orient="records") == [ + {"s": "a", "__left_count__": 2}, + {"s": "b", "__left_count__": 1}, + ] + cached = _connected_join_cached_singleton_dst_source_counts( + graph, + graph._edges, + x_edges, + {"kind": "x"}, + "z", + src_col="s", + dst_col="d", + engine=Engine.PANDAS, + cache_store=cache_store, + ) + assert cached is counts + assert _connected_join_cached_singleton_dst_source_counts( + graph, + graph._edges, + x_edges, + {"kind": {"nested": "not-cacheable"}}, + "z", + src_col="s", + dst_col="d", + engine=Engine.PANDAS, + cache_store=cache_store, + ) is None + + filtered_nodes = _filter_nodes_for_fast_count( + pd.DataFrame({"id": ["a", "b"], "kind": ["keep", "drop"]}), + {"kind": "keep"}, + engine=Engine.PANDAS, + ) + assert filtered_nodes.to_dict(orient="records") == [{"id": "a", "kind": "keep"}] + + def test_connected_join_post_property_columns_walks_nested_params() -> None: plan = ConnectedMatchJoinPlan( pattern_chains=(), @@ -19348,6 +19751,135 @@ def test_connected_join_post_property_columns_walks_nested_params() -> None: assert _connected_join_post_property_columns(plan, "b") == ["age"] +def test_lowering_coverage_helper_projection_and_aggregate_decisions() -> None: + alias_targets = {"a": ASTNode(name="a"), "b": ASTNode(name="b")} + + assert _projection_ref_from_expr_safe(" a.city ", alias_targets) == ("a", "city") + assert _projection_ref_from_expr_safe("a.city.name", alias_targets) is None + assert _projection_ref_from_expr_safe("missing.city", alias_targets) is None + assert _projection_ref_from_expr_safe("a.bad-name", alias_targets) is None + + mixed_query = parse_cypher("MATCH (a)-->(b) RETURN a.city + count(b) AS score") + assert _clause_has_mixed_aggregate_item(mixed_query, alias_targets=alias_targets, params=None) + + split_query = parse_cypher("MATCH (a)-->(b) RETURN a.city AS city, count(b) AS n") + assert not _clause_has_mixed_aggregate_item(split_query, alias_targets=alias_targets, params=None) + assert _binding_prop_alias_set(split_query, alias_targets=alias_targets, params=None) == ["a"] + + count_only_query = parse_cypher("MATCH (a)-->(b) RETURN count(*) AS n") + assert _binding_prop_alias_set(count_only_query, alias_targets=alias_targets, params=None) == [] + + +def test_lowering_coverage_helper_conservative_binding_branches() -> None: + alias_targets = {"a": ASTNode(name="a"), "b": ASTNode(name="b")} + + star_query = parse_cypher("MATCH (a) RETURN *") + assert not _clause_has_mixed_aggregate_item(star_query, alias_targets=alias_targets, params=None) + assert _binding_prop_alias_set(star_query, alias_targets={"a": ASTNode(name="a")}, params=None) == [] + + bad_mixed_query = parse_cypher("MATCH (a) RETURN ghost.city + count(a) AS score") + assert _clause_has_mixed_aggregate_item(bad_mixed_query, alias_targets={"a": ASTNode(name="a")}, params=None) + + list_aggregate_query = parse_cypher("MATCH (a) RETURN [count(a)] AS xs") + assert _clause_has_mixed_aggregate_item(list_aggregate_query, alias_targets={"a": ASTNode(name="a")}, params=None) + + with_query = parse_cypher("MATCH (a) WITH a RETURN a.city AS city") + assert _binding_prop_alias_set(with_query, alias_targets={"a": ASTNode(name="a")}, params=None) is None + + optional_query = parse_cypher("OPTIONAL MATCH (a)-->(b) RETURN a.city AS city") + assert _binding_prop_alias_set(optional_query, alias_targets=alias_targets, params=None) is None + + missing_alias_query = parse_cypher("MATCH (a) RETURN ghost.city AS g") + assert _binding_prop_alias_set(missing_alias_query, alias_targets={"a": ASTNode(name="a")}, params=None) == [] + assert _binding_prop_alias_set(missing_alias_query, alias_targets={}, params=None) is None + + collect_query = parse_cypher("MATCH (a) RETURN collect(a.city) AS cities") + assert _binding_prop_alias_set(collect_query, alias_targets={"a": ASTNode(name="a")}, params=None) is None + + aggregate_prop_query = parse_cypher("MATCH (a)-->(b) RETURN avg(b.age) AS avg_age") + assert _binding_prop_alias_set(aggregate_prop_query, alias_targets=alias_targets, params=None) == ["b"] + + order_by_query = parse_cypher("MATCH (a)-->(b) RETURN count(*) AS n ORDER BY b.age") + assert _binding_prop_alias_set(order_by_query, alias_targets=alias_targets, params=None) == ["b"] + + +def test_lowering_coverage_helper_exception_fallbacks(monkeypatch: Any) -> None: + alias_targets = {"a": ASTNode(name="a")} + query = parse_cypher("MATCH (a) RETURN a.city AS city") + + def raise_validation(*args: Any, **kwargs: Any) -> None: + raise GFQLValidationError(ErrorCode.E201, "forced validation failure") + + with monkeypatch.context() as m: + m.setattr(lowering_module, "_parse_row_expr", raise_validation) + assert _clause_has_mixed_aggregate_item(query, alias_targets=alias_targets, params=None) + + with monkeypatch.context() as m: + m.setattr(lowering_module, "_match_pattern_elements", lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("forced"))) + assert _binding_prop_alias_set(query, alias_targets=alias_targets, params=None) is None + + with monkeypatch.context() as m: + m.setattr(lowering_module, "_collect_aggregate_specs_for_clause", lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("forced"))) + assert _binding_prop_alias_set(query, alias_targets=alias_targets, params=None) is None + + with monkeypatch.context() as m: + m.setattr(lowering_module, "_expr_match_alias_usage", raise_validation) + assert _binding_prop_alias_set(query, alias_targets=alias_targets, params=None) is None + + +def test_lowering_coverage_helper_direct_margin_paths() -> None: + span = SourceSpan(1, 1, 1, 1, 0, 0) + alias_targets = {"a": ASTNode(name="a"), "r": ASTEdgeForward(name="r")} + + assert _render_row_where_operand_text(ParameterRef("p", span)) == "$p" + assert _render_row_where_operand_text(None) == "null" + assert _render_row_where_operand_text(True) == "true" + assert _render_row_where_operand_text(7) == "7" + assert _row_where_predicate_text(WherePredicate(PropertyRef("a", "age", span), "contains", "x", span)) is None + assert _row_where_predicate_text(WherePredicate(LabelRef("a", ("Person",), span), "has_labels", None, span)) is None + + node_spec = _AggregateSpec("count(DISTINCT a)", "n", "count", "a", True, 1, 1) + edge_spec = _AggregateSpec("count(DISTINCT r)", "n", "count", "r", True, 1, 1) + collect_edge_spec = _AggregateSpec("collect(DISTINCT r)", "rs", "collect", "r", True, 1, 1) + avg_distinct_spec = _AggregateSpec("avg(DISTINCT a.age)", "avg_age", "avg", "a.age", True, 1, 1) + + assert _distinct_aggregate_expr_text(_AggregateSpec("count(*)", "n", "count", None, True, 1, 1), alias_targets=alias_targets) is None + assert _aggregate_runtime_spec(node_spec, alias_targets=alias_targets) == ("count_distinct", NODE_IDENTITY_COLUMN) + assert _aggregate_runtime_spec(edge_spec, alias_targets=alias_targets) == ("count_distinct", "__gfql_edge_index_0__") + with pytest.raises(GFQLValidationError): + _aggregate_runtime_spec(collect_edge_spec, alias_targets=alias_targets) + with pytest.raises(GFQLValidationError): + _aggregate_runtime_spec(avg_distinct_spec, alias_targets=alias_targets) + + assert _whole_row_group_entity_expr("a", alias_targets=alias_targets, field="return", line=1, column=1) == "__node_entity__(a)" + assert _whole_row_group_entity_expr("r", alias_targets=alias_targets, field="return", line=1, column=1) == "__edge_entity__(r)" + assert _whole_row_group_key_expr("r", alias_targets=alias_targets, field="return", line=1, column=1) == "__gfql_edge_index_0__" + with pytest.raises(GFQLValidationError): + _whole_row_group_key_expr("missing", alias_targets=alias_targets, field="return", line=1, column=1) + + multi_agg_query = parse_cypher("MATCH (a)-[r]->(b) RETURN count(a) AS ca, count(b) AS cb") + with pytest.raises(GFQLValidationError): + _active_match_alias_for_stage( + unwinds=(), + clause=multi_agg_query.return_, + order_by_clause=multi_agg_query.order_by, + alias_targets={"a": ASTNode(name="a"), "b": ASTNode(name="b"), "r": ASTEdgeForward(name="r")}, + params=None, + ) + + boundary_exc = GFQLValidationError(ErrorCode.E108, "forced", field="return", value=["a", "b"]) + assert _is_multi_source_match_alias_boundary_error(boundary_exc, alias_targets={"a": ASTNode(name="a"), "b": ASTNode(name="b")}) + assert not _is_multi_source_match_alias_boundary_error(GFQLValidationError(ErrorCode.E201, "forced", field="return", value=["a", "b"]), alias_targets={"a": ASTNode(name="a"), "b": ASTNode(name="b")}) + + assert _expr_property_access_node_aliases( + "[a.city, r.weight]", + alias_targets=alias_targets, + params=None, + field="return", + line=1, + column=1, + ) == {"a", "r"} + def test_string_cypher_parenthesized_and_preserves_missing_property_as_null() -> None: # A parenthesized AND must stay on the row-filter path, not the filter_dict path: # the row engine treats an absent property as null (Cypher semantics), whereas diff --git a/graphistry/tests/compute/gfql/cypher/test_row_pushdown.py b/graphistry/tests/compute/gfql/cypher/test_row_pushdown.py index 90466a5361..41095a3e84 100644 --- a/graphistry/tests/compute/gfql/cypher/test_row_pushdown.py +++ b/graphistry/tests/compute/gfql/cypher/test_row_pushdown.py @@ -19,8 +19,7 @@ import graphistry import graphistry.compute.gfql.cypher.lowering as _lowering from graphistry.compute.ast import ASTCall -from graphistry.compute.exceptions import GFQLTypeError, GFQLValidationError -from graphistry.compute.gfql.call.validation import validate_call_params +from graphistry.compute.exceptions import GFQLValidationError from graphistry.compute.gfql.cypher.api import cypher_to_gfql from graphistry.compute.gfql.cypher.row_pushdown import ( _flatten_and_conjuncts, @@ -74,11 +73,6 @@ def g(): PUSHDOWN_QUERIES: List[Tuple[str, str]] = [ ("edge_search", "MATCH (a)-[e]->(b) WHERE searchAny(e, 'WIRE') RETURN e.w AS w ORDER BY w LIMIT 5000"), ("node_search", "MATCH (n) WHERE searchAny(n, 'Ember') RETURN n.id AS id ORDER BY id LIMIT 5000"), - ( - "node_search_explicit_columns", - "MATCH (n) WHERE searchAny(n, 'Ember', {columns: ['name']}) " - "RETURN n.id AS id ORDER BY id LIMIT 5000", - ), ( "panel", "MATCH (a)-[e]->(b) WHERE a.kind <> 'internal' AND NOT (a.flag = true AND a.score < 0.1) " @@ -209,29 +203,3 @@ def test_pass_is_noop_on_plain_chain(): from graphistry.compute.chain import Chain ch = Chain([ASTCall("rows", {"table": "nodes"}), ASTCall("limit", {"value": 10})]) assert apply_row_prefilter_pushdown(ch) is ch - - -def test_alias_prefilter_wire_options_require_real_bools(): - valid = { - "n": [{ - "kind": "search_any", - "term": "Ember", - "case_sensitive": False, - "regex": True, - "columns": ["name"], - }] - } - assert validate_call_params("rows", {"alias_prefilters": valid}) == { - "alias_prefilters": valid - } - - for key in ("case_sensitive", "regex"): - invalid = {"n": [{"kind": "search_any", "term": "Ember", key: "false"}]} - with pytest.raises(GFQLTypeError): - validate_call_params("rows", {"alias_prefilters": invalid}) - - -def test_alias_prefilter_wire_rejects_present_null_columns(): - invalid = {"n": [{"kind": "search_any", "term": "Ember", "columns": None}]} - with pytest.raises(GFQLTypeError): - validate_call_params("rows", {"alias_prefilters": invalid}) diff --git a/graphistry/tests/compute/gfql/index/test_index.py b/graphistry/tests/compute/gfql/index/test_index.py index a48dbb97b1..c8a48bf216 100644 --- a/graphistry/tests/compute/gfql/index/test_index.py +++ b/graphistry/tests/compute/gfql/index/test_index.py @@ -197,8 +197,6 @@ def test_explain_exposes_planner_diagnostics(graph, engine): free Σ-degree fanout estimate (from CSR group_offsets), chosen direction, the cost-gate threshold, and a human-readable decision_reason — not just a used-index yes/no. This is the EXPLAIN enrichment the indexing roadmap calls for.""" - from graphistry.compute.gfql.index import cost_gate_min_frontier - chain = [n({"id": 0}), e_forward(hops=1)] # force → index path taken → per-step + top-level diagnostics populated rep = graph.gfql_explain(chain, index_policy="force", engine=engine) @@ -207,14 +205,10 @@ def test_explain_exposes_planner_diagnostics(graph, engine): assert isinstance(rep["est_result_rows"], int) and rep["est_result_rows"] >= 0 assert "index" in (rep["decision_reason"] or ""), rep["decision_reason"] step = next(s for s in rep["steps"] if s.get("path") == "index") - for k in ( - "frontier_n", "n_keys", "seed_deg_sum", "est_result_rows", - "threshold_frac", "min_frontier_floor", - ): + for k in ("frontier_n", "n_keys", "seed_deg_sum", "est_result_rows", "threshold_frac"): assert k in step, (k, step) assert step["est_result_rows"] == step["seed_deg_sum"] # est == free Σ-degree assert step["seed_deg_sum"] >= 0 - assert step["min_frontier_floor"] == cost_gate_min_frontier() # off (index resident) → the planner records *why* it scanned, not just that it did gi = graph.gfql_index_all(engine=engine) @@ -330,89 +324,6 @@ def test_cost_gate_frac_tuning(monkeypatch): reset_cost_gate_frac() -@pytest.mark.parametrize("engine", ENGINES) -def test_cost_gate_small_frontier_floor_indexes_tiny_slices(engine): - """#1726 follow-up: a small seeded frontier on a small-n_keys slice (e.g. an SNB - per-edge-type homogeneous frame) must take the resident index on every engine. - Before the floor, the frac gate (0.02 on polars/cudf) tripped on any slice with - n_keys <= 50 even for a single seed -> O(E) scan with an O(degree) index resident. - Result is identical either way (the gate is a pure routing heuristic).""" - from graphistry.compute.gfql.index import index_trace - rng = np.random.default_rng(3) - n_src = 30 # n_keys ~30: frac gate alone would demand frontier < 0.6 on polars/cudf - edf = pd.DataFrame({ - "src": rng.integers(0, n_src, 600), - "dst": rng.integers(0, 2000, 600), - }) - g = graphistry.edges(edf, "src", "dst").materialize_nodes() - gi = g.gfql_index_all(engine=engine) - seeds = pd.DataFrame({"id": edf["src"].iloc[:1].astype("int64")}) # frontier_n=1 - with index_trace() as steps: - out = gi.hop(nodes=seeds, engine=engine, hops=1, direction="forward") - assert any(s.get("path") == "index" for s in steps), (engine, steps) - scan_out = g.hop(nodes=seeds, engine=engine, hops=1, direction="forward") - assert _sig(out) == _sig(scan_out), engine # correctness path-independent - - # A frontier just past the floor on the same tiny slice still bails to scan - # (frac gate: 17 >= 0.02*30 and 17 >= 0.5*30) -> the bulk-hop protection stands. - many = pd.DataFrame({"id": np.arange(17, dtype="int64")}) - with index_trace() as steps2: - out2 = gi.hop(nodes=many, engine=engine, hops=1, direction="forward") - assert not any(s.get("path") == "index" for s in steps2), (engine, steps2) - assert any("scan cheaper" in (s.get("decision_reason") or "") for s in steps2), (engine, steps2) - scan_out2 = g.hop(nodes=many, engine=engine, hops=1, direction="forward") - assert _sig(out2) == _sig(scan_out2), engine - - -def test_cost_gate_min_frontier_tuning(monkeypatch): - from graphistry.compute.gfql.index import cost_gate_min_frontier - from graphistry.compute.gfql.index.cost import _COST_GATE_MIN_FRONTIER_DEFAULT - - monkeypatch.delenv("GFQL_INDEX_COST_GATE_MIN_FRONTIER", raising=False) - assert cost_gate_min_frontier() == _COST_GATE_MIN_FRONTIER_DEFAULT == 16 - - monkeypatch.setenv("GFQL_INDEX_COST_GATE_MIN_FRONTIER", "32") - assert cost_gate_min_frontier() == 32 - - monkeypatch.setenv("GFQL_INDEX_COST_GATE_MIN_FRONTIER", "0") # 0 disables the floor - assert cost_gate_min_frontier() == 0 - - monkeypatch.setenv("GFQL_INDEX_COST_GATE_MIN_FRONTIER", "-1") - with pytest.raises(ValueError, match="GFQL_INDEX_COST_GATE_MIN_FRONTIER"): - cost_gate_min_frontier() - monkeypatch.setenv("GFQL_INDEX_COST_GATE_MIN_FRONTIER", "abc") - with pytest.raises(ValueError, match="GFQL_INDEX_COST_GATE_MIN_FRONTIER"): - cost_gate_min_frontier() - - -def test_cost_gate_min_frontier_floor_disabled_restores_frac_gate(monkeypatch): - """Floor=0 restores pure frac gating: a 1-seed hop on a tiny-n_keys slice bails - again under a sub-frac (proves the floor is what routes it to the index).""" - from graphistry.Engine import Engine - from graphistry.compute.gfql.index import ( - index_trace, reset_cost_gate_frac, set_cost_gate_frac, - ) - monkeypatch.delenv("GFQL_INDEX_COST_GATE_MIN_FRONTIER", raising=False) - edf = pd.DataFrame({"src": [0, 1, 2, 0, 1], "dst": [10, 11, 12, 13, 14]}) - g = graphistry.edges(edf, "src", "dst").materialize_nodes() - gi = g.gfql_index_all(engine="pandas") - seeds = pd.DataFrame({"id": [0]}) - set_cost_gate_frac(Engine.PANDAS, 0.02) # emulate the vectorized-engine frac on CPU - try: - # frac alone would bail (1 >= 0.02*3): the floor keeps the index engaged - with index_trace() as steps: - gi.hop(nodes=seeds, engine="pandas", hops=1, direction="forward") - assert any(s.get("path") == "index" for s in steps), steps - # disable the floor -> the frac gate governs again -> scan - monkeypatch.setenv("GFQL_INDEX_COST_GATE_MIN_FRONTIER", "0") - with index_trace() as steps2: - gi.hop(nodes=seeds, engine="pandas", hops=1, direction="forward") - assert not any(s.get("path") == "index" for s in steps2), steps2 - assert any("scan cheaper" in (s.get("decision_reason") or "") for s in steps2), steps2 - finally: - reset_cost_gate_frac() - - def test_column_mismatch_raises_not_silent(graph): # A custom column that doesn't match the binding must raise, not silently no-op. with pytest.raises(NotImplementedError): diff --git a/graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py b/graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py index b60d3c2402..10b9703e43 100644 --- a/graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py +++ b/graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py @@ -182,3 +182,78 @@ def test_polars_binding_rows_undirected_self_loop(): rpl.sort_values(key).reset_index(drop=True), check_dtype=False, ) + + + +def test_polars_binding_rows_focused_native_coverage(): + """Focused coverage for narrow native-polars helpers used by binding rows.""" + from graphistry.Engine import Engine + from graphistry.compute.dataframe.join import ( + connected_inner_join_rows, + joined_alias_columns, + joined_hidden_scalar_columns, + ) + from graphistry.compute.gfql.lazy.engine.polars.row_pipeline import ( + binding_rows_polars, + can_order_by_native, + can_select_native, + select_extend_polars, + ) + + # Polars dataframe join helper paths used by connected comma joins. + hidden = pl.DataFrame({ + "a.__gfql_hidden_score": [None, 2], + "b.__gfql_hidden_score": [1, None], + }) + hidden_out = joined_hidden_scalar_columns(hidden) + assert hidden_out.select("__gfql_hidden_score").to_series().to_list() == [1, 2] + + alias_out = joined_alias_columns(pl.DataFrame({"a.id": ["a1"], "b.b": ["b1"]})) + assert alias_out.select(["a", "b"]).to_dicts() == [{"a": "a1", "b": "b1"}] + + joined = connected_inner_join_rows( + pl.DataFrame({"a.id": ["a1", "a2"], "a.num": [1, 2]}), + pl.DataFrame({"a.id": ["a1", "a1", "a3"], "b.id": ["b1", "b2", "b3"]}), + join_cols=["a.id"], + keep_cols=["a.id", "b.id"], + engine=Engine.POLARS, + ) + assert joined.select(["a.id", "b.id"]).to_dicts() == [ + {"a.id": "a1", "b.id": "b1"}, + {"a.id": "a1", "b.id": "b2"}, + ] + + # select_extend_polars is the binding-aggregate extension helper, distinct from + # public with_(extend=True) dispatch. + g = graphistry.nodes(pl.from_pandas(NODES), "id").edges(pl.from_pandas(EDGES), "s", "d") + extended = select_extend_polars(g, [("age2", "age + 1")]) + assert extended is not None + assert extended._nodes.select(["id", "age2"]).to_dicts()[0] == {"id": 0, "age2": 11} + assert select_extend_polars(g, [("bad", "unsupported(age)")]) is None + + assert can_select_native([("age2", "age + 1")], ["age"]) + assert can_order_by_native([("age", "desc")], ["age"]) + + +def test_polars_binding_rows_decline_branches_direct(): + """Directly exercise honest-decline branches that are hard to reach via Cypher.""" + from graphistry.compute.gfql.lazy.engine.polars.row_pipeline import binding_rows_polars + + bo = serialize_binding_ops([n(name="a"), e_forward(), n(name="b")]) + no_edges = graphistry.nodes(pl.from_pandas(NODES), "id") + assert binding_rows_polars(no_edges, bo) is None + + seeded = graphistry.nodes(pl.from_pandas(NODES), "id").edges(pl.from_pandas(EDGES), "s", "d") + setattr(seeded, "_gfql_start_nodes", pl.DataFrame({"id": [0]})) + assert binding_rows_polars(seeded, bo) is None + + g = graphistry.nodes(pl.from_pandas(NODES), "id").edges(pl.from_pandas(EDGES), "s", "d") + assert binding_rows_polars(g, serialize_binding_ops([n(name="a"), n(name="b")])) is None + assert binding_rows_polars(g, serialize_binding_ops([n(name="a", query="id > 0"), e_forward(), n(name="b")])) is None + assert binding_rows_polars(g, serialize_binding_ops([n(name="a"), e_forward(source_node_match={"kind": "a"}), n(name="b")])) is None + assert binding_rows_polars(g, serialize_binding_ops([n(name="a"), e_forward(label_seeds=True), n(name="b")])) is None + + # Polars cannot implicitly unify mismatched join-key dtypes; decline on collect SchemaError. + bad_edges = pl.DataFrame({"s": ["0"], "d": ["1"]}) + bad = graphistry.nodes(pl.from_pandas(NODES), "id").edges(bad_edges, "s", "d") + assert binding_rows_polars(bad, bo) is None diff --git a/graphistry/tests/compute/test_engine_coercion.py b/graphistry/tests/compute/test_engine_coercion.py index 853591603b..574900673e 100644 --- a/graphistry/tests/compute/test_engine_coercion.py +++ b/graphistry/tests/compute/test_engine_coercion.py @@ -356,48 +356,6 @@ def test_polars_idempotent(self): self.assertIs(twice._edges, once._edges) -class TestPlNanToNull(NoAuthTestCase): - """_pl_nan_to_null must be identity-stable when no NaN is present (#1726) while - preserving the NaN -> null conversion when a float column actually carries a NaN.""" - - @unittest.skipUnless(HAS_POLARS, "polars not installed") - def test_float_col_without_nan_returns_same_object(self): - # A NaN-free float column must NOT trigger a rebuild (frame-identity churn would - # defeat the #1658 CSR index, which keys resident indexes on `source_ref is df`). - from graphistry.Engine import _pl_nan_to_null - inp = pl.DataFrame({"src": ["a", "b"], "w": [1.0, 2.0]}) - out = _pl_nan_to_null(inp) - self.assertIs(out, inp) - - @unittest.skipUnless(HAS_POLARS, "polars not installed") - def test_no_float_cols_returns_same_object(self): - from graphistry.Engine import _pl_nan_to_null - inp = pl.DataFrame({"src": ["a", "b"], "dst": ["b", "c"]}) - out = _pl_nan_to_null(inp) - self.assertIs(out, inp) - - @unittest.skipUnless(HAS_POLARS, "polars not installed") - def test_float_col_with_nan_still_converts_to_null(self): - # NaN-present path is unchanged: NaN -> null, other values untouched. - from graphistry.Engine import _pl_nan_to_null - inp = pl.DataFrame({"src": ["a", "b", "c"], "w": [1.0, float("nan"), 3.0]}) - out = _pl_nan_to_null(inp) - self.assertIsNot(out, inp) - w = out.get_column("w") - # The NaN became a genuine null (not still a NaN), non-NaN values preserved. - self.assertEqual(w.null_count(), 1) - self.assertFalse(bool(w.is_nan().any())) - self.assertEqual(w.to_list(), [1.0, None, 3.0]) - - @unittest.skipUnless(HAS_POLARS, "polars not installed") - def test_existing_nulls_without_nan_returns_same_object(self): - # A float column with real nulls but no NaN needs no rewrite (identity preserved). - from graphistry.Engine import _pl_nan_to_null - inp = pl.DataFrame({"src": ["a", "b"], "w": [1.0, None]}) - out = _pl_nan_to_null(inp) - self.assertIs(out, inp) - - class TestToPandas(NoAuthTestCase): """to_pandas() method — converts all supported input types to pandas."""