From 7f5e438dcbe99ac9677e74602590965b22fabdb6 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 18 Jul 2026 20:04:41 -0700 Subject: [PATCH 1/7] fix(gfql): ORDER BY DESC places NULLs first (openCypher NULL-largest) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit F2 from the OLAP-stack review. The general row pipeline hardcoded nulls-last for every ORDER BY key on all engines, so `ORDER BY DESC` over a column with NULLs was mis-ordered and `DESC ... LIMIT k` silently returned the wrong top-k (dropping the NULL group, which openCypher ranks first since NULL is largest). - pandas/cuDF (row/pipeline.py): per-key null-indicator single-pass sort (ind = col.isna(), sorted in the key's own direction) — stability-free so cuDF's non-stable quicksort is safe; scalar na_position cannot be per-key. - polars (order_by_polars): per-key nulls_last list aligned with exprs. - test_order_by_null_placement.py: hand-derived openCypher oracle, ASC+DESC x grouped+scalar, across pandas/polars/cuDF. Mutation-verified vs pristine master (wrong [C,B] -> correct [None,C]) on all three engines. Pre-existing on master (not introduced by the OLAP split); the OLAP fast paths already ordered nulls correctly, so this aligns the general/fallback path. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL --- CHANGELOG.md | 3 + .../gfql/lazy/engine/polars/row_pipeline.py | 12 ++- graphistry/compute/gfql/row/pipeline.py | 21 ++++- .../cypher/test_order_by_null_placement.py | 87 +++++++++++++++++++ 4 files changed, 118 insertions(+), 5 deletions(-) create mode 100644 graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 95b0306882..87de78db1f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,9 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ### Documentation - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine ` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator). +### Fixed +- **GFQL Cypher `ORDER BY ... DESC` now places NULLs FIRST (openCypher NULL-largest), fixing silent-wrong `DESC ... LIMIT k` top-k**: openCypher orders NULL as the largest value (`ASC` → nulls last, `DESC` → nulls first), but the general row pipeline hardcoded nulls-last for every key on all engines — so any `ORDER BY DESC` over a column containing NULLs was mis-ordered and `DESC ... LIMIT k` silently dropped the NULL group that should rank first. Fixed on pandas/cuDF (per-key null-indicator single-pass in `row/pipeline.py`; stability-free, so cuDF's non-stable sort is safe) and polars (per-key `nulls_last` list in `order_by_polars`). Hand-derived-oracle regression suite `test_order_by_null_placement.py` across pandas/polars/cuDF. Pre-existing bug (not introduced by the OLAP split); the OLAP fast paths already ordered nulls correctly, so this brings the general/fallback path into agreement. + ### Added - **GFQL Cypher row-pipeline single-alias predicate pushdown**: Planner-generated `rows(alias_prefilters=...)` hints pre-filter eligible node and single-hop edge alias frames before binding joins on pandas/cuDF while retaining the original post-join filters for exact semantics and engine-safe fallback. - **GFQL viz-filter-pipeline acceptance suite + regression benchmark (viz-filter L3)**: `test_viz_pipeline_conformance.py` — curated full-panel pipelines (node+edge filters, exclusion-dominates composition, `(pred OR IS NULL)` keep-null leaves, both EXISTS prune-isolated flavors, searchAny composition, deterministic paging), graph-state prune shapes with exact node+edge pins, a 40-seed panel-state fuzzer with an independent plain-pandas second oracle, and a case/regex/unicode trick matrix (ß/İ full-case-mapping pins, metachar literal-vs-regex, null cells, Categorical) — all parity-or-NIE across pandas/cuDF/polars/polars-gpu. `benchmarks/gfql/viz_filter_pipeline.py` — six streamgl-viz panel scenarios (filters, keep-self GRAPH prune, EXISTS prune, node/edge search, combined) at 100K/1M/10M with native-frame-per-engine fairness, an NIE-tolerant matrix, and JSON receipts (first receipt: 100k × 4 engines, everything within the ~350ms interactive reference except pandas combined). Documented findings from the first runs: the same-path WHERE route dedupes parallel edges (diverges from the panel algebra's edge multiplicity — pinned + tracked), and edge-alias searchAny declines on polars (tracked). diff --git a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py index c7c33a940d..f7acbda112 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py +++ b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py @@ -706,9 +706,15 @@ def order_by_polars(g: Plottable, keys: Sequence[Any]) -> Optional[Plottable]: if lowered is None: return None exprs, descending = lowered - # cypher ORDER BY puts NULLs last; polars defaults to nulls_last=False (pandas sort_values - # default puts NaN last only for asc), so set nulls_last=True to match pandas na_position='last'. - return _rewrap(g, table.sort(exprs, descending=descending, nulls_last=True)) + # openCypher orders NULL as the LARGEST value: ASC -> nulls last, DESC -> nulls FIRST. + # (Previously hardcoded nulls_last=True, which mis-ordered DESC keys and silently returned + # the wrong `... DESC LIMIT k` top-k over a column containing NULLs.) polars accepts a + # per-key nulls_last list aligned with exprs. + if isinstance(descending, bool): + nulls_last: Any = not descending + else: + nulls_last = [not d for d in descending] + return _rewrap(g, table.sort(exprs, descending=descending, nulls_last=nulls_last)) # Native aggs: count/sum/avg/min/max/count_distinct/collect/collect_distinct; stdev/percentile diff --git a/graphistry/compute/gfql/row/pipeline.py b/graphistry/compute/gfql/row/pipeline.py index 836bbf3097..d713e9b6f1 100644 --- a/graphistry/compute/gfql/row/pipeline.py +++ b/graphistry/compute/gfql/row/pipeline.py @@ -4744,8 +4744,25 @@ def order_by(self, keys: List[Any]) -> "Plottable": if sort_cols: try: - effective_sort_cols = sort_cols + [row_order_col] - effective_ascending = ascending + [True] + # openCypher orders NULL as the LARGEST value: ASC -> nulls last, DESC -> nulls + # FIRST. pandas/cuDF sort_values `na_position` is a SINGLE scalar (cannot be + # per-key) and cuDF has no stable sort, so use a per-key null-indicator column + # sorted in the SAME direction as its key: ind = col.isna(); for an ASC key + # (ascending=True) True (null) sorts LAST, for a DESC key (ascending=False) True + # sorts FIRST. Matches the OLAP fast-path fix and order_by_polars. (Previously + # sort_values defaulted na_position='last' for every key, silently returning the + # wrong `... DESC LIMIT k` top-k over a column containing NULLs.) + interleaved_cols: List[Any] = [] + interleaved_ascending: List[bool] = [] + for _i, (_sc, _asc) in enumerate(zip(sort_cols, ascending)): + _ind_col = RowPipelineMixin._gfql_fresh_col_name( + work_df.columns, f"__gfql_sort_nullrank_{_i}__" + ) + work_df = work_df.assign(**{_ind_col: work_df[_sc].isna()}) + interleaved_cols.extend([_ind_col, _sc]) + interleaved_ascending.extend([_asc, _asc]) + effective_sort_cols = interleaved_cols + [row_order_col] + effective_ascending = interleaved_ascending + [True] out_df = work_df.sort_values(by=effective_sort_cols, ascending=effective_ascending) except Exception as exc: raise ValueError( diff --git a/graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py b/graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py new file mode 100644 index 0000000000..469b9edd4a --- /dev/null +++ b/graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py @@ -0,0 +1,87 @@ +"""Regression: openCypher NULL ordering in the general row-pipeline ORDER BY. + +openCypher orders NULL as the LARGEST value: + - ``ORDER BY x ASC`` -> NULLs LAST + - ``ORDER BY x DESC`` -> NULLs FIRST + +The general row pipeline previously hardcoded NULLs-last for every key (pandas/cuDF +``sort_values`` defaulting ``na_position='last'``; polars ``order_by_polars`` passing +``nulls_last=True``), so any ``ORDER BY DESC`` over a column containing NULLs was +mis-ordered on BOTH pandas and polars and ``DESC ... LIMIT k`` silently returned the wrong +top-k (it dropped the NULL group, which should rank first). Oracle here is HAND-DERIVED +openCypher truth, NOT parity-vs-parent (the parity suites had no NULL in the ordered key, +which is how this survived). +""" +import pandas as pd +import pytest + +import graphistry + +try: + import cudf # noqa: F401 + _HAS_CUDF = True +except Exception: # pragma: no cover - depends on test env + _HAS_CUDF = False + +_ENGINES = ["pandas", "polars"] + (["cudf"] if _HAS_CUDF else []) + +# city groups: A:3 (0,3,7) B:2 (1,4) None:2 (2,5) C:1 (6) +_NODES = pd.DataFrame( + {"id": [0, 1, 2, 3, 4, 5, 6, 7], "city": ["A", "B", None, "A", "B", None, "C", "A"]} +) +_EDGES = pd.DataFrame({"s": [0, 1], "d": [1, 2], "eid": [0, 1]}) + + +def _g(): + return graphistry.nodes(_NODES, "id").edges(_EDGES, "s", "d").bind(edge="eid") + + +def _recs(nodes): + """Rows as list-of-dicts with NaN/pd.NA normalized to None (engine-agnostic).""" + if hasattr(nodes, "to_pandas"): + nodes = nodes.to_pandas() + out = [] + for rec in nodes.to_dict("records"): + out.append({k: (None if pd.isna(v) else v) for k, v in rec.items()}) + return out + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_grouped_desc_limit_puts_null_group_first(engine): + # openCypher DESC: NULL largest -> [None(2), C(1), B(2), A(3)]; LIMIT 2 -> [None, C]. + r = _g().gfql( + "MATCH (m) RETURN m.city AS city, count(m) AS c ORDER BY city DESC LIMIT 2", + engine=engine, + )._nodes + assert _recs(r) == [{"city": None, "c": 2}, {"city": "C", "c": 1}] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_grouped_asc_limit_keeps_null_group_last(engine): + # openCypher ASC: NULLs last -> [A(3), B(2), C(1), None(2)]; LIMIT 2 -> [A, B]. + r = _g().gfql( + "MATCH (m) RETURN m.city AS city, count(m) AS c ORDER BY city ASC LIMIT 2", + engine=engine, + )._nodes + assert _recs(r) == [{"city": "A", "c": 3}, {"city": "B", "c": 2}] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_scalar_desc_orders_nulls_first(engine): + # DESC over scalar column: NULL rows first, then C, B, A (stable within group by id). + r = _g().gfql( + "MATCH (m) RETURN m.id AS id, m.city AS city ORDER BY city DESC", + engine=engine, + )._nodes + cities = [rec["city"] for rec in _recs(r)] + assert cities == [None, None, "C", "B", "B", "A", "A", "A"] + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_scalar_asc_orders_nulls_last(engine): + r = _g().gfql( + "MATCH (m) RETURN m.id AS id, m.city AS city ORDER BY city ASC", + engine=engine, + )._nodes + cities = [rec["city"] for rec in _recs(r)] + assert cities == ["A", "A", "A", "B", "B", "C", None, None] From 19979cac3107a517544a96da8c0f4a4e420a439e Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 18 Jul 2026 20:09:44 -0700 Subject: [PATCH 2/7] fix(gfql): from_json registers comparison predicates, not numeric-only Bug #1 from the OLAP-stack review. The predicate from_json registry mapped EQ/NE/GT/GE/LT/LE/Between/IsNA/NotNA to predicates.numeric (numeric-only), but Cypher lowering and the public predicate API build predicates.comparison (which also accept strings + temporals). Same __name__ meant a JSON round-trip downgraded a live comparison predicate to numeric-only, raising GFQLTypeError "val must be numeric" for a string/temporal equality or a temporal comparison (breaks remote GFQL, serialization, caching). Fix: from_json imports the comparison versions. Numeric-valued predicates are unaffected (comparison is a superset). Round-trip regression tests added to test_from_json.py (string/temporal/numeric eq + temporal gt), mutation-verified (revert import -> 10 tests fail). Full predicate suite 139 passed/0 failed on GPU. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL --- CHANGELOG.md | 1 + graphistry/compute/predicates/from_json.py | 7 ++- .../compute/predicates/test_from_json.py | 43 +++++++++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 87de78db1f..c30a0a77d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine ` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator). ### Fixed +- **GFQL predicate `from_json` no longer downgrades comparison predicates to numeric-only (fixes JSON round-trip of string/temporal `=`, `<`, `>`, etc.)**: the predicate deserialization registry mapped `EQ`/`NE`/`GT`/`GE`/`LT`/`LE`/`Between` to the numeric-only `predicates.numeric` classes, while Cypher lowering and the public predicate API build the richer `predicates.comparison` versions (which also accept strings and temporals). A JSON round-trip (remote GFQL, serialization, caching) therefore rebuilt a string/temporal equality or a temporal comparison as numeric-only, raising `GFQLTypeError "val must be numeric"`. `from_json` now registers the `comparison` versions; numeric-valued predicates are unaffected (comparison is a superset). Round-trip regression tests in `test_from_json.py`. - **GFQL Cypher `ORDER BY ... DESC` now places NULLs FIRST (openCypher NULL-largest), fixing silent-wrong `DESC ... LIMIT k` top-k**: openCypher orders NULL as the largest value (`ASC` → nulls last, `DESC` → nulls first), but the general row pipeline hardcoded nulls-last for every key on all engines — so any `ORDER BY DESC` over a column containing NULLs was mis-ordered and `DESC ... LIMIT k` silently dropped the NULL group that should rank first. Fixed on pandas/cuDF (per-key null-indicator single-pass in `row/pipeline.py`; stability-free, so cuDF's non-stable sort is safe) and polars (per-key `nulls_last` list in `order_by_polars`). Hand-derived-oracle regression suite `test_order_by_null_placement.py` across pandas/polars/cuDF. Pre-existing bug (not introduced by the OLAP split); the OLAP fast paths already ordered nulls correctly, so this brings the general/fallback path into agreement. ### Added diff --git a/graphistry/compute/predicates/from_json.py b/graphistry/compute/predicates/from_json.py index c4ec6e26d3..7b4637bbe0 100644 --- a/graphistry/compute/predicates/from_json.py +++ b/graphistry/compute/predicates/from_json.py @@ -4,7 +4,12 @@ from graphistry.compute.predicates.categorical import Duplicated from graphistry.compute.predicates.is_in import IsIn from graphistry.compute.predicates.logical import AllOf -from graphistry.compute.predicates.numeric import GT, LT, GE, LE, EQ, NE, Between, IsNA, NotNA +# Register the comparison.py versions (numeric + temporal + string), NOT numeric.py's numeric-only +# classes: they share __name__ ("EQ", "GT", ...) so the from_json type registry would otherwise +# downgrade a live comparison predicate to numeric-only on a JSON round-trip — raising "val must be +# numeric" for a string/temporal equality or temporal comparison built by Cypher lowering / the +# public predicate API (both of which use comparison.py). See from_json round-trip regression tests. +from graphistry.compute.predicates.comparison import GT, LT, GE, LE, EQ, NE, Between, IsNA, NotNA from graphistry.compute.predicates.str import ( Contains, Startswith, Endswith, Match, Fullmatch, IsNumeric, IsAlpha, IsDecimal, IsDigit, IsLower, IsUpper, IsSpace, IsAlnum, IsTitle, IsNull, NotNull diff --git a/graphistry/tests/compute/predicates/test_from_json.py b/graphistry/tests/compute/predicates/test_from_json.py index 3eb9a17cb8..306af32053 100644 --- a/graphistry/tests/compute/predicates/test_from_json.py +++ b/graphistry/tests/compute/predicates/test_from_json.py @@ -191,3 +191,46 @@ def test_json_roundtrip_validates(): with pytest.raises(GFQLTypeError): json_data_invalid = {'type': 'Startswith', 'pat': 'test', 'case': 'not_bool', 'na': None} from_json(json_data_invalid) + + +# --- Regression: comparison predicates must round-trip as the comparison.py (numeric+temporal+ +# string) versions, NOT be downgraded to numeric.py's numeric-only classes. Cypher lowering and +# the public predicate API build comparison.* predicates; from_json previously registered the +# same-named numeric.* classes, so a JSON round-trip lost temporal/string capability and RAISED +# "val must be numeric" for a string/temporal equality or a temporal comparison. --- +import datetime as _dt + +from graphistry.compute.predicates.comparison import ( + eq as _cmp_eq, gt as _cmp_gt, ge as _cmp_ge, lt as _cmp_lt, le as _cmp_le, ne as _cmp_ne, + EQ as _CmpEQ, GT as _CmpGT, +) + + +@pytest.mark.parametrize("factory,val", [ + (_cmp_eq, "foo"), # string equality — previously RAISED on round-trip + (_cmp_eq, 5), # numeric equality + (_cmp_eq, _dt.date(2020, 1, 1)), # temporal equality — previously RAISED + (_cmp_ne, "bar"), + (_cmp_gt, _dt.date(2020, 1, 1)), # temporal comparison — previously RAISED + (_cmp_gt, 3), + (_cmp_ge, 3), + (_cmp_lt, 3), + (_cmp_le, 3), +]) +def test_comparison_predicate_json_roundtrip_preserves_comparison_class(factory, val): + pred = factory(val) + back = from_json(pred.to_json()) + # must be the SAME comparison.py class, not a numeric.py downgrade + assert type(back) is type(pred), ( + f"{type(pred).__module__}.{type(pred).__name__} round-tripped to " + f"{type(back).__module__}.{type(back).__name__}" + ) + assert type(back).__module__ == "graphistry.compute.predicates.comparison" + + +def test_string_and_temporal_eq_survive_roundtrip_without_raising(): + # The exact pre-fix failure: numeric.EQ.validate() raised GFQLTypeError "val must be numeric". + from_json(_cmp_eq("foo").to_json()) # no raise + from_json(_cmp_eq(_dt.date(2020, 1, 1)).to_json()) # no raise + assert isinstance(from_json({'type': 'EQ', 'val': 'foo'}), _CmpEQ) + assert isinstance(from_json({'type': 'GT', 'val': {'type': 'date', 'value': '2020-01-01'}}), _CmpGT) From cfa8f9dfd307d4fa813a098fd2ac7a4182a65722 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 18 Jul 2026 20:15:06 -0700 Subject: [PATCH 3/7] fix(gfql): string predicates value-safe on non-string columns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bug #2 from the OLAP-stack review. Contains/Startswith/Endswith/Match/Fullmatch called `s.str.` directly, raising an opaque AttributeError ("Can only use .str accessor with string values!") on a numeric/temporal/bool column (pandas and cuDF). Fix: guard each predicate — a non-string-capable series yields an all-null result (openCypher: string op over a non-string is null -> excluded), mirroring the per-cell behavior on an object column holding non-strings. Never stringifies (would diverge pandas<->cuDF, e.g. wrongly match `5 CONTAINS '5'`). String, mixed-object, and all-null columns unchanged. Helpers _series_supports_str_ops / _nonstring_null_result. Regression tests in test_str.py across dtypes; cuDF parity confirmed (int -> all-null, not stringified). str suite 106 passed. Scope: the 5 pattern predicates. The str-property _CallablePredicate family (IsNumeric/IsAlpha/...) is deferred — its semantics on non-string dtypes are genuinely ambiguous, tracked for a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL --- CHANGELOG.md | 1 + graphistry/compute/predicates/str.py | 44 +++++++++++++++++++ .../tests/compute/predicates/test_str.py | 33 ++++++++++++++ 3 files changed, 78 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c30a0a77d1..0c8b274618 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine ` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator). ### Fixed +- **GFQL string predicates (`Contains`/`Startswith`/`Endswith`/`Match`/`Fullmatch`) are now value-safe on non-string columns**: applying a string predicate to a numeric/temporal/bool column raised an opaque `AttributeError: Can only use .str accessor with string values!` on pandas and cuDF. They now follow openCypher semantics — a string op over a non-string value is null → excluded (matching the established per-cell behavior on an object column holding non-strings) — and never stringify the column (which would diverge pandas↔cuDF, e.g. wrongly matching `5 CONTAINS '5'`). String, mixed-object, and all-null columns are unchanged. Regression tests in `test_str.py` across dtypes; cuDF parity confirmed. - **GFQL predicate `from_json` no longer downgrades comparison predicates to numeric-only (fixes JSON round-trip of string/temporal `=`, `<`, `>`, etc.)**: the predicate deserialization registry mapped `EQ`/`NE`/`GT`/`GE`/`LT`/`LE`/`Between` to the numeric-only `predicates.numeric` classes, while Cypher lowering and the public predicate API build the richer `predicates.comparison` versions (which also accept strings and temporals). A JSON round-trip (remote GFQL, serialization, caching) therefore rebuilt a string/temporal equality or a temporal comparison as numeric-only, raising `GFQLTypeError "val must be numeric"`. `from_json` now registers the `comparison` versions; numeric-valued predicates are unaffected (comparison is a superset). Round-trip regression tests in `test_from_json.py`. - **GFQL Cypher `ORDER BY ... DESC` now places NULLs FIRST (openCypher NULL-largest), fixing silent-wrong `DESC ... LIMIT k` top-k**: openCypher orders NULL as the largest value (`ASC` → nulls last, `DESC` → nulls first), but the general row pipeline hardcoded nulls-last for every key on all engines — so any `ORDER BY DESC` over a column containing NULLs was mis-ordered and `DESC ... LIMIT k` silently dropped the NULL group that should rank first. Fixed on pandas/cuDF (per-key null-indicator single-pass in `row/pipeline.py`; stability-free, so cuDF's non-stable sort is safe) and polars (per-key `nulls_last` list in `order_by_polars`). Hand-derived-oracle regression suite `test_order_by_null_placement.py` across pandas/polars/cuDF. Pre-existing bug (not introduced by the OLAP split); the OLAP fast paths already ordered nulls correctly, so this brings the general/fallback path into agreement. diff --git a/graphistry/compute/predicates/str.py b/graphistry/compute/predicates/str.py index 3d5d858d90..2263d13a9d 100644 --- a/graphistry/compute/predicates/str.py +++ b/graphistry/compute/predicates/str.py @@ -7,6 +7,44 @@ from graphistry.compute.typing import SeriesT +def _series_supports_str_ops(s: Any) -> bool: + """True if ``s`` has a usable string accessor (object / pandas ``string`` / categorical-of-str). + + A numeric, temporal, or boolean column does NOT: pandas and cuDF both raise on ``s.str`` + attribute access for those dtypes. Used to make the string predicates value-safe instead of + surfacing an opaque ``AttributeError: Can only use .str accessor with string values!``. + """ + try: + s.str # the accessor validates dtype on attribute access + except (AttributeError, TypeError): + return False + return True + + +def _nonstring_null_result(s: Any, na: Optional[bool]) -> Any: + """Result of a string predicate applied to a NON-string column. + + openCypher treats a string operation over a non-string value as null (a non-string value is + not a string, so the predicate is unknown → null → excluded from a ``WHERE``). This mirrors the + established per-cell behavior on an *object* column holding non-strings (those cells already + become null), and — crucially — NEVER stringifies the column (which would diverge pandas↔cuDF + and wrongly match, e.g. ``5 CONTAINS '5'``). ``na`` overrides the fill only when the caller + pinned it (``na=True``/``na=False``). + """ + fill = None if na is None else na + n = len(s) + is_cudf = hasattr(s, '__module__') and 'cudf' in s.__module__ + if is_cudf: + import cudf + out = cudf.Series([fill] * n) + try: + out.index = s.index + except Exception: + pass + return out + return pd.Series([fill] * n, index=getattr(s, 'index', None)) + + def _cudf_mask_value(result: Any, mask: Any, value: Any) -> Any: try: return result.mask(mask, value) @@ -91,6 +129,8 @@ def __init__( self.regex = regex def __call__(self, s: SeriesT) -> SeriesT: + if not _series_supports_str_ops(s): + return _nonstring_null_result(s, self.na) is_cudf = hasattr(s, '__module__') and 'cudf' in s.__module__ # workaround cuDF not supporting 'case' and 'na' parameters @@ -260,6 +300,8 @@ def _compute_result(self, s: SeriesT, is_cudf: bool) -> SeriesT: return self._match_boundary(s, self.pat) def __call__(self, s: SeriesT) -> SeriesT: + if not _series_supports_str_ops(s): + return _nonstring_null_result(s, self.na) is_cudf = hasattr(s, '__module__') and 'cudf' in s.__module__ result = self._compute_result(s, is_cudf) if is_cudf: @@ -381,6 +423,8 @@ def _compute_result(self, s: SeriesT, is_cudf: bool) -> SeriesT: raise NotImplementedError def __call__(self, s: SeriesT) -> SeriesT: + if not _series_supports_str_ops(s): + return _nonstring_null_result(s, self.na) is_cudf = hasattr(s, '__module__') and 'cudf' in s.__module__ result = self._compute_result(s, is_cudf) if is_cudf: diff --git a/graphistry/tests/compute/predicates/test_str.py b/graphistry/tests/compute/predicates/test_str.py index ffcdb16beb..0faaba70ce 100644 --- a/graphistry/tests/compute/predicates/test_str.py +++ b/graphistry/tests/compute/predicates/test_str.py @@ -931,3 +931,36 @@ def test_case_crossing_ranges_and_nonascii_decline(self): assert _cudf_casefold_or_decline("[a-c]") == "[a-c]" # same-case range folds fine assert _cudf_casefold_or_decline("[0-9]+") == "[0-9]+" # non-letter range folds fine assert _cudf_casefold_or_decline("e-MAIL") == "e-mail" # class-free literal hyphen folds fine + + +# --- Regression: string predicates must be VALUE-SAFE on non-string columns. Previously +# `s.str.contains(...)` etc. raised `AttributeError: Can only use .str accessor with string +# values!` on a numeric/temporal/bool column. openCypher: a string op over a non-string value is +# null -> excluded (matching the per-cell behavior on an object column holding non-strings), and +# must NOT stringify (which would diverge pandas<->cuDF, e.g. wrongly match `5 CONTAINS '5'`). --- +from graphistry.compute.predicates.str import Contains, Startswith, Endswith, Match, Fullmatch + + +@pytest.mark.parametrize("pred", [ + Contains("1", regex=False), Startswith("1"), Endswith("1"), + Match("1"), Fullmatch("1"), +]) +@pytest.mark.parametrize("series", [ + pd.Series([1, 15, 3]), # int + pd.Series([1.0, 15.0, 3.0]), # float + pd.Series([True, False, True]), # bool +]) +def test_string_predicate_on_nonstring_column_is_null_not_raise(pred, series): + # Must not raise, and must be all-null (excluded), never stringified. + result = pred(series) + assert len(result) == len(series) + assert all(pd.isna(v) for v in result), f"{pred!r} on {series.dtype} -> {list(result)}" + + +def test_string_predicate_on_string_and_mixed_unchanged(): + # String dtype: normal matching still works. + assert list(Contains("a", regex=False)(pd.Series(["abc", "xyz", None]))) == [True, False, None] + # Mixed object: non-string cells already become null (unchanged behavior). + r = Contains("a", regex=False)(pd.Series(["a1", 2, None], dtype="object")) + assert r.iloc[0] == True # noqa: E712 + assert pd.isna(r.iloc[1]) and pd.isna(r.iloc[2]) From b232b9422a736cf6d797d61d6b49ed872d4aae34 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 18 Jul 2026 20:26:15 -0700 Subject: [PATCH 4/7] fix(gfql): polars raises clean GFQLSchemaError for string predicate on non-string column MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bugs #3 (polars connected-join clean-NIE) + #4 (pushdown polars-dtype) from the OLAP-stack review. A string predicate (CONTAINS/STARTS WITH/ENDS WITH/regex) on a Categorical/Enum/numeric column raised a clean GFQLSchemaError on pandas/cuDF but leaked polars' opaque InvalidOperationError ("expected String type, got: cat"). filter_by_dict_polars now raises the same GFQLSchemaError (E302) so all three engines agree; categorical is treated as non-string exactly as filter_by_dict. Investigation (documented): the connected-join predicate-pushdown dtype gate is parity-correct on polars for string/numeric/eq/bool columns (pushdown_probe: PARITY=True on all) — no silent-wrong push found; the only divergence was this opaque-vs-clean error, now fixed. Regression test test_polars_string_predicate_nonstring.py; polars conformance 406 passed/0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL --- CHANGELOG.md | 1 + .../gfql/lazy/engine/polars/predicates.py | 19 ++++++- .../test_polars_string_predicate_nonstring.py | 49 +++++++++++++++++++ 3 files changed, 68 insertions(+), 1 deletion(-) create mode 100644 graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c8b274618..2dd1a52ea2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine ` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator). ### Fixed +- **GFQL polars engine raises a clean typed error (not an opaque polars `InvalidOperationError`) for a string predicate on a non-string column**: `WHERE col STARTS WITH/CONTAINS/regex ...` on a Categorical/Enum/numeric column raised a clean `GFQLSchemaError` ("string predicate used on non-string column") on pandas/cuDF but leaked polars' internal `InvalidOperationError: expected String type, got: cat` on polars. `filter_by_dict_polars` now raises the same `GFQLSchemaError` (categorical treated as non-string, exactly as `filter_by_dict`), so all three engines agree. Regression test `test_polars_string_predicate_nonstring.py`; also confirms connected-join predicate pushdown is parity-correct on polars for string/numeric/eq/bool columns. - **GFQL string predicates (`Contains`/`Startswith`/`Endswith`/`Match`/`Fullmatch`) are now value-safe on non-string columns**: applying a string predicate to a numeric/temporal/bool column raised an opaque `AttributeError: Can only use .str accessor with string values!` on pandas and cuDF. They now follow openCypher semantics — a string op over a non-string value is null → excluded (matching the established per-cell behavior on an object column holding non-strings) — and never stringify the column (which would diverge pandas↔cuDF, e.g. wrongly matching `5 CONTAINS '5'`). String, mixed-object, and all-null columns are unchanged. Regression tests in `test_str.py` across dtypes; cuDF parity confirmed. - **GFQL predicate `from_json` no longer downgrades comparison predicates to numeric-only (fixes JSON round-trip of string/temporal `=`, `<`, `>`, etc.)**: the predicate deserialization registry mapped `EQ`/`NE`/`GT`/`GE`/`LT`/`LE`/`Between` to the numeric-only `predicates.numeric` classes, while Cypher lowering and the public predicate API build the richer `predicates.comparison` versions (which also accept strings and temporals). A JSON round-trip (remote GFQL, serialization, caching) therefore rebuilt a string/temporal equality or a temporal comparison as numeric-only, raising `GFQLTypeError "val must be numeric"`. `from_json` now registers the `comparison` versions; numeric-valued predicates are unaffected (comparison is a superset). Round-trip regression tests in `test_from_json.py`. - **GFQL Cypher `ORDER BY ... DESC` now places NULLs FIRST (openCypher NULL-largest), fixing silent-wrong `DESC ... LIMIT k` top-k**: openCypher orders NULL as the largest value (`ASC` → nulls last, `DESC` → nulls first), but the general row pipeline hardcoded nulls-last for every key on all engines — so any `ORDER BY DESC` over a column containing NULLs was mis-ordered and `DESC ... LIMIT k` silently dropped the NULL group that should rank first. Fixed on pandas/cuDF (per-key null-indicator single-pass in `row/pipeline.py`; stability-free, so cuDF's non-stable sort is safe) and polars (per-key `nulls_last` list in `order_by_polars`). Hand-derived-oracle regression suite `test_order_by_null_placement.py` across pandas/polars/cuDF. Pre-existing bug (not introduced by the OLAP split); the OLAP fast paths already ordered nulls correctly, so this brings the general/fallback path into agreement. diff --git a/graphistry/compute/gfql/lazy/engine/polars/predicates.py b/graphistry/compute/gfql/lazy/engine/polars/predicates.py index bdbf734a51..5bad911bd8 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/predicates.py +++ b/graphistry/compute/gfql/lazy/engine/polars/predicates.py @@ -13,7 +13,7 @@ from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Union from graphistry.compute.predicates.ASTPredicate import ASTPredicate -from graphistry.compute.predicates.str import Fullmatch, Match +from graphistry.compute.predicates.str import Contains, Endswith, Fullmatch, Match, Startswith from graphistry.compute.filter_by_dict import resolve_filter_column from .dtypes import is_numeric as _dtype_numeric, is_stringlike as _dtype_stringlike @@ -325,6 +325,23 @@ def filter_by_dict_polars(df: "pl.DataFrame", filter_dict: "Optional[Dict[str, A f"comparison on column {resolved_col!r}; use engine='pandas' for this " f"query (no pandas fallback; parity-or-error by design)" ) + # String predicate on a non-`String` column: pandas/cuDF raise a clean + # GFQLSchemaError (E302) at schema validation, but polars would build `.str.` and + # raise an OPAQUE `InvalidOperationError` ("expected String type, got: cat") at collect + # on a Categorical/Enum/numeric column. Raise the SAME clean, typed error so all three + # engines agree (categorical is treated as non-string here, exactly as filter_by_dict). + if isinstance(resolved_val, (Contains, Startswith, Endswith, Match)): + _col_dtype = df.schema.get(resolved_col) + if _col_dtype is not None and _col_dtype != pl.String: + from graphistry.compute.exceptions import ErrorCode, GFQLSchemaError + raise GFQLSchemaError( + ErrorCode.E302, + f'Type mismatch: string predicate used on non-string column "{resolved_col}"', + field=col, + value=f"{resolved_val.__class__.__name__}(...)", + column_type=str(_col_dtype), + suggestion='Use numeric predicates like gt() or lt() for numeric columns', + ) expr = predicate_to_expr(resolved_col, resolved_val, df.schema.get(resolved_col)) if expr is None: # decline (NIE): no native lowering for this predicate; no pandas bridge. diff --git a/graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py b/graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py new file mode 100644 index 0000000000..3b9fd43c31 --- /dev/null +++ b/graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py @@ -0,0 +1,49 @@ +"""Regression: a string predicate on a non-String polars column must raise the SAME clean +GFQLSchemaError that pandas/cuDF raise — not an opaque polars `InvalidOperationError` +("expected String type, got: cat"). Categorical/Enum are treated as non-string here, exactly +as `filter_by_dict` does on pandas/cuDF, so the three engines stay consistent. +""" +import pandas as pd +import pytest + +import graphistry +from graphistry.compute.exceptions import GFQLSchemaError + +try: + import cudf # noqa: F401 + _HAS_CUDF = True +except Exception: # pragma: no cover + _HAS_CUDF = False + +_ENGINES = ["pandas", "polars"] + (["cudf"] if _HAS_CUDF else []) + +_N = pd.DataFrame({ + "id": [0, 1, 2, 3, 4], + "cat": pd.Categorical(["Alice", "Bob", "Amy", None, "Al"]), + "age": [30, 25, 40, 22, None], +}) +_E = pd.DataFrame({"s": [0, 1, 2, 3], "d": [1, 2, 3, 4], "eid": [0, 1, 2, 3]}) + + +def _g(): + return graphistry.nodes(_N, "id").edges(_E, "s", "d").bind(edge="eid") + + +@pytest.mark.parametrize("engine", _ENGINES) +@pytest.mark.parametrize("query", [ + "MATCH (m) WHERE m.cat STARTS WITH 'A' RETURN m.id AS id", # categorical + "MATCH (m) WHERE m.cat CONTAINS 'm' RETURN m.id AS id", # categorical + "MATCH (m) WHERE m.age CONTAINS '2' RETURN m.id AS id", # numeric +]) +def test_string_predicate_on_nonstring_column_raises_schema_error_all_engines(engine, query): + with pytest.raises(GFQLSchemaError): + _g().gfql(query, engine=engine)._nodes + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_equality_on_categorical_still_works(engine): + # `=` on a categorical is fine on all engines (not a `.str` op). + r = _g().gfql("MATCH (m) WHERE m.cat = 'Bob' RETURN m.id AS id", engine=engine)._nodes + if hasattr(r, "to_pandas"): + r = r.to_pandas() + assert r.to_dict("records") == [{"id": 1}] From c08f16e606413ede77d6dd9b83aec6e283d2fd2c Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 18 Jul 2026 20:34:53 -0700 Subject: [PATCH 5/7] fix(gfql): CASE with mixed-dtype branches is engine-consistent on cuDF Bug #6 (R4) from the OLAP-stack review. pandas coerces CASE branches to a common type, but cuDF's `.where` raises TypeError "cudf does not support mixed types", surfaced as a hard GFQLTypeError. Bit `CASE WHEN path IS NULL THEN -1 ELSE length(path) END` over an UNREACHABLE shortestPath (int -1 vs object/null hops branch): pandas returned -1, cuDF errored. Fix: the row-AST CASE evaluator (_gfql_eval_expr_ast) now catches the cuDF mixed-type TypeError, unifies the branches via _gfql_coerce_case_branch_dtypes (common numeric dtype, then string fallback), and retries -> cuDF returns the same value pandas does. Reachable paths keep clean int parity (direct .where succeeds, no coercion). Validated on #1705 (shortestPath): reachable 0->2 = 2 both engines; unreachable 0->4 = -1 (pandas) / -1.0 (cuDF, value-equal, no raise). General regression test test_case_mixed_dtype.py (all-null object column, same mechanism, no shortestPath) mutation-verified: pristine master cuDF RAISES, fixed cuDF returns -1. CASE test suite 58 passed. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL --- CHANGELOG.md | 1 + graphistry/compute/gfql/row/pipeline.py | 28 ++++++++++++- .../compute/gfql/test_case_mixed_dtype.py | 41 +++++++++++++++++++ 3 files changed, 69 insertions(+), 1 deletion(-) create mode 100644 graphistry/tests/compute/gfql/test_case_mixed_dtype.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 2dd1a52ea2..ab039031f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine ` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator). ### Fixed +- **GFQL Cypher `CASE` with mixed-dtype branches is now engine-consistent on cuDF (no more `GFQLTypeError`)**: pandas coerces the two `CASE` branches to a common type, but cuDF's `.where` raised `TypeError: cudf does not support mixed types`, surfaced as a hard `GFQLTypeError`. This bit `CASE WHEN path IS NULL THEN -1 ELSE length(path) END` over an UNREACHABLE `shortestPath` (int `-1` branch vs an object/null hops branch) — pandas returned `-1`, cuDF errored. The row-AST `CASE` evaluator now unifies the branch dtypes and retries, so cuDF returns the same value pandas does. Regression test `test_case_mixed_dtype.py`. - **GFQL polars engine raises a clean typed error (not an opaque polars `InvalidOperationError`) for a string predicate on a non-string column**: `WHERE col STARTS WITH/CONTAINS/regex ...` on a Categorical/Enum/numeric column raised a clean `GFQLSchemaError` ("string predicate used on non-string column") on pandas/cuDF but leaked polars' internal `InvalidOperationError: expected String type, got: cat` on polars. `filter_by_dict_polars` now raises the same `GFQLSchemaError` (categorical treated as non-string, exactly as `filter_by_dict`), so all three engines agree. Regression test `test_polars_string_predicate_nonstring.py`; also confirms connected-join predicate pushdown is parity-correct on polars for string/numeric/eq/bool columns. - **GFQL string predicates (`Contains`/`Startswith`/`Endswith`/`Match`/`Fullmatch`) are now value-safe on non-string columns**: applying a string predicate to a numeric/temporal/bool column raised an opaque `AttributeError: Can only use .str accessor with string values!` on pandas and cuDF. They now follow openCypher semantics — a string op over a non-string value is null → excluded (matching the established per-cell behavior on an object column holding non-strings) — and never stringify the column (which would diverge pandas↔cuDF, e.g. wrongly matching `5 CONTAINS '5'`). String, mixed-object, and all-null columns are unchanged. Regression tests in `test_str.py` across dtypes; cuDF parity confirmed. - **GFQL predicate `from_json` no longer downgrades comparison predicates to numeric-only (fixes JSON round-trip of string/temporal `=`, `<`, `>`, etc.)**: the predicate deserialization registry mapped `EQ`/`NE`/`GT`/`GE`/`LT`/`LE`/`Between` to the numeric-only `predicates.numeric` classes, while Cypher lowering and the public predicate API build the richer `predicates.comparison` versions (which also accept strings and temporals). A JSON round-trip (remote GFQL, serialization, caching) therefore rebuilt a string/temporal equality or a temporal comparison as numeric-only, raising `GFQLTypeError "val must be numeric"`. `from_json` now registers the `comparison` versions; numeric-valued predicates are unaffected (comparison is a superset). Round-trip regression tests in `test_from_json.py`. diff --git a/graphistry/compute/gfql/row/pipeline.py b/graphistry/compute/gfql/row/pipeline.py index d713e9b6f1..8382060fbc 100644 --- a/graphistry/compute/gfql/row/pipeline.py +++ b/graphistry/compute/gfql/row/pipeline.py @@ -240,6 +240,22 @@ def _gfql_fresh_col_name(columns: Any, prefix: str) -> str: col = f"{col}_x" return col + @staticmethod + def _gfql_coerce_case_branch_dtypes(a: Any, b: Any) -> "Tuple[Any, Any]": + """Cast two CASE branches to a common dtype for engines (cuDF) whose ``.where`` rejects + mismatched dtypes, where pandas coerces. Prefer a numeric common type (covers int / float / + null — the shortestPath hops branch); fall back to string; last resort return unchanged so + the caller re-raises the original error rather than masking an unexpected failure.""" + for target in ("float64",): + try: + return a.astype(target), b.astype(target) + except Exception: + continue + try: + return a.astype("str"), b.astype("str") + except Exception: + return a, b + @staticmethod def _gfql_mask_fill(series: Any, mask: Any, value: Any) -> Any: out = series.copy() @@ -1648,7 +1664,17 @@ def _floor_series(s: Any) -> Any: cond_mask = self._gfql_bool_mask(table_df, cond_value) cond_null = self._gfql_null_mask(table_df, cond_value) cond_true = cond_mask & ~cond_null - return True, true_value.where(cond_true, false_value) + try: + return True, true_value.where(cond_true, false_value) + except TypeError: + # cuDF rejects `.where` across mismatched branch dtypes ("cudf does not support + # mixed types"), where pandas coerces to a common type. This surfaced as a hard + # GFQLTypeError for e.g. `CASE WHEN path IS NULL THEN -1 ELSE length(path) END` + # over an UNREACHABLE shortestPath (the hops branch is an object/null column while + # the other branch is int -1). Unify the two branches to a common dtype and retry so + # CASE is engine-consistent (cuDF returns -1 like pandas instead of raising). + tv, fv = RowPipelineMixin._gfql_coerce_case_branch_dtypes(true_value, false_value) + return True, tv.where(cond_true, fv) if isinstance(node, QuantifierExpr): source_ok, list_value = self._gfql_eval_expr_ast(table_df, node.source) diff --git a/graphistry/tests/compute/gfql/test_case_mixed_dtype.py b/graphistry/tests/compute/gfql/test_case_mixed_dtype.py new file mode 100644 index 0000000000..8cdca3fc8f --- /dev/null +++ b/graphistry/tests/compute/gfql/test_case_mixed_dtype.py @@ -0,0 +1,41 @@ +"""Regression: CASE with branches of different dtypes must be engine-consistent. + +pandas coerces the two CASE branches to a common type, but cuDF's `.where` raises +`TypeError: cudf does not support mixed types`, which surfaced as a hard +`GFQLTypeError` (e.g. `CASE WHEN path IS NULL THEN -1 ELSE length(path) END` over an +UNREACHABLE shortestPath: the hops branch is an object/null column, the other is int +-1). The evaluator now unifies the branch dtypes and retries, so cuDF returns the same +value pandas does instead of raising. This general repro uses an all-null object column +(same mechanism, no shortestPath dependency). +""" +import pandas as pd +import pytest + +import graphistry + +try: + import cudf # noqa: F401 + _HAS_CUDF = True +except Exception: # pragma: no cover + _HAS_CUDF = False + +_ENGINES = ["pandas"] + (["cudf"] if _HAS_CUDF else []) + +_N = pd.DataFrame({"id": [0, 1, 2], "x": pd.Series([None, None, None], dtype="object")}) +_E = pd.DataFrame({"s": [0], "d": [1], "eid": [0]}) + + +@pytest.mark.parametrize("engine", _ENGINES) +def test_case_mixed_dtype_branches_do_not_raise(engine): + # int THEN branch vs all-null object ELSE branch: cuDF `.where` used to raise + # "cudf does not support mixed types"; must now return -1 for every row like pandas. + g = graphistry.nodes(_N, "id").edges(_E, "s", "d").bind(edge="eid") + r = g.gfql( + "MATCH (m) RETURN m.id AS id, CASE WHEN m.x IS NULL THEN -1 ELSE m.x END AS r ORDER BY id", + engine=engine, + )._nodes + if hasattr(r, "to_pandas"): + r = r.to_pandas() + # value-equal across engines (cuDF coerces to float -1.0; pandas keeps int -1) + assert [rec["id"] for rec in r.to_dict("records")] == [0, 1, 2] + assert all(float(rec["r"]) == -1.0 for rec in r.to_dict("records")) From fc1937f64bc2f5539ae1fbb34fcd47a89c28765a Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 18 Jul 2026 20:45:03 -0700 Subject: [PATCH 6/7] test(gfql): guard polars-parametrized tests behind polars availability The new order-by-null and polars-string-predicate regression tests hardcoded engine="polars", which ImportErrors on the CPU CI lanes (test-gfql-core / test-core-python) that ship without polars. Guard polars like cudf (_HAS_POLARS), so those lanes run pandas(+cudf) and the test-polars lane runs polars. No behavior change where polars is installed. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL --- .../compute/gfql/cypher/test_order_by_null_placement.py | 8 +++++++- .../gfql/test_polars_string_predicate_nonstring.py | 8 +++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py b/graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py index 469b9edd4a..cb9ed701fa 100644 --- a/graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py +++ b/graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py @@ -17,13 +17,19 @@ import graphistry +try: + import polars # noqa: F401 + _HAS_POLARS = True +except Exception: # pragma: no cover - depends on test env + _HAS_POLARS = False + try: import cudf # noqa: F401 _HAS_CUDF = True except Exception: # pragma: no cover - depends on test env _HAS_CUDF = False -_ENGINES = ["pandas", "polars"] + (["cudf"] if _HAS_CUDF else []) +_ENGINES = ["pandas"] + (["polars"] if _HAS_POLARS else []) + (["cudf"] if _HAS_CUDF else []) # city groups: A:3 (0,3,7) B:2 (1,4) None:2 (2,5) C:1 (6) _NODES = pd.DataFrame( diff --git a/graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py b/graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py index 3b9fd43c31..cf12113726 100644 --- a/graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py +++ b/graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py @@ -9,13 +9,19 @@ import graphistry from graphistry.compute.exceptions import GFQLSchemaError +try: + import polars # noqa: F401 + _HAS_POLARS = True +except Exception: # pragma: no cover + _HAS_POLARS = False + try: import cudf # noqa: F401 _HAS_CUDF = True except Exception: # pragma: no cover _HAS_CUDF = False -_ENGINES = ["pandas", "polars"] + (["cudf"] if _HAS_CUDF else []) +_ENGINES = ["pandas"] + (["polars"] if _HAS_POLARS else []) + (["cudf"] if _HAS_CUDF else []) _N = pd.DataFrame({ "id": [0, 1, 2, 3, 4], From d8816e611d21ec98b50c715b6a702b4d49a44ea3 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 18 Jul 2026 21:08:38 -0700 Subject: [PATCH 7/7] test(gfql): satisfy changed-line-coverage for the engine-parity fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The changed-line-coverage gate (combines core + gfql + polars lanes; NO cuDF lane) flagged the cuDF-only branches and two polars lines. - cuDF-only branches marked `# pragma: no cover` with rationale (structurally uncoverable by this gate — no cuDF lane; all validated on dgx + mutation- verified): str.py _nonstring_null_result cuDF branch, pipeline.py CASE mixed-dtype except + _gfql_coerce_case_branch_dtypes helper. - order_by_polars: restructured the scalar-vs-list `descending` branch into a single coverable path (normalize to a per-key list); multi-key ORDER BY validated parity pandas/polars on dgx. - Added the new polars regression tests (test_polars_string_predicate_nonstring, test_order_by_null_placement) to bin/test-polars.sh so the polars lane actually exercises (and covers) the GFQLSchemaError raise + null-order paths. No production behavior change beyond the order_by_polars refactor (parity- validated). 24 order-by/string-predicate tests pass incl. polars. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL --- bin/test-polars.sh | 2 ++ .../compute/gfql/lazy/engine/polars/row_pipeline.py | 12 +++++------- graphistry/compute/gfql/row/pipeline.py | 4 ++-- graphistry/compute/predicates/str.py | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/bin/test-polars.sh b/bin/test-polars.sh index 6d7965cee6..24ab0a8933 100755 --- a/bin/test-polars.sh +++ b/bin/test-polars.sh @@ -19,6 +19,8 @@ POLARS_TEST_FILES=( graphistry/tests/compute/gfql/test_engine_polars_row_pipeline.py graphistry/tests/compute/gfql/test_engine_polars_cypher_conformance.py graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py + graphistry/tests/compute/gfql/test_polars_string_predicate_nonstring.py + graphistry/tests/compute/gfql/cypher/test_order_by_null_placement.py graphistry/tests/compute/gfql/test_conformance_ledger.py # index tests exercise the seeded-index hook in the polars hop entry (hop.py) — without # them the hook dominates the now-thin file and trips its per-file coverage floor diff --git a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py index f7acbda112..4671d76f8c 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py +++ b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py @@ -708,13 +708,11 @@ def order_by_polars(g: Plottable, keys: Sequence[Any]) -> Optional[Plottable]: exprs, descending = lowered # openCypher orders NULL as the LARGEST value: ASC -> nulls last, DESC -> nulls FIRST. # (Previously hardcoded nulls_last=True, which mis-ordered DESC keys and silently returned - # the wrong `... DESC LIMIT k` top-k over a column containing NULLs.) polars accepts a - # per-key nulls_last list aligned with exprs. - if isinstance(descending, bool): - nulls_last: Any = not descending - else: - nulls_last = [not d for d in descending] - return _rewrap(g, table.sort(exprs, descending=descending, nulls_last=nulls_last)) + # the wrong `... DESC LIMIT k` top-k over a column containing NULLs.) Normalize `descending` + # to a per-key list (it may arrive as a scalar bool) so `nulls_last` mirrors it per key. + descending_list = descending if isinstance(descending, list) else [descending] * len(exprs) + nulls_last = [not d for d in descending_list] + return _rewrap(g, table.sort(exprs, descending=descending_list, nulls_last=nulls_last)) # Native aggs: count/sum/avg/min/max/count_distinct/collect/collect_distinct; stdev/percentile diff --git a/graphistry/compute/gfql/row/pipeline.py b/graphistry/compute/gfql/row/pipeline.py index 8382060fbc..0f9eedd2e5 100644 --- a/graphistry/compute/gfql/row/pipeline.py +++ b/graphistry/compute/gfql/row/pipeline.py @@ -241,7 +241,7 @@ def _gfql_fresh_col_name(columns: Any, prefix: str) -> str: return col @staticmethod - def _gfql_coerce_case_branch_dtypes(a: Any, b: Any) -> "Tuple[Any, Any]": + def _gfql_coerce_case_branch_dtypes(a: Any, b: Any) -> "Tuple[Any, Any]": # pragma: no cover - only reached from the cuDF CASE-mixed-dtype retry above; no cuDF coverage lane (validated on dgx) """Cast two CASE branches to a common dtype for engines (cuDF) whose ``.where`` rejects mismatched dtypes, where pandas coerces. Prefer a numeric common type (covers int / float / null — the shortestPath hops branch); fall back to string; last resort return unchanged so @@ -1666,7 +1666,7 @@ def _floor_series(s: Any) -> Any: cond_true = cond_mask & ~cond_null try: return True, true_value.where(cond_true, false_value) - except TypeError: + except TypeError: # pragma: no cover - cuDF-only (.where mixed-dtype); no cuDF lane in the coverage gate (validated on dgx) # cuDF rejects `.where` across mismatched branch dtypes ("cudf does not support # mixed types"), where pandas coerces to a common type. This surfaced as a hard # GFQLTypeError for e.g. `CASE WHEN path IS NULL THEN -1 ELSE length(path) END` diff --git a/graphistry/compute/predicates/str.py b/graphistry/compute/predicates/str.py index 2263d13a9d..10a547c9eb 100644 --- a/graphistry/compute/predicates/str.py +++ b/graphistry/compute/predicates/str.py @@ -34,7 +34,7 @@ def _nonstring_null_result(s: Any, na: Optional[bool]) -> Any: fill = None if na is None else na n = len(s) is_cudf = hasattr(s, '__module__') and 'cudf' in s.__module__ - if is_cudf: + if is_cudf: # pragma: no cover - cuDF-only; the changed-line-coverage gate has no cuDF lane (validated on dgx) import cudf out = cudf.Series([fill] * n) try: