Skip to content

spec: grouped operators crash on string labels under pandas 3.0 (ArrowStringArray) #924

Description

@FabianHofmann

Note

This bug report was drafted by an AI agent from a reproduced failure. The
reproduction, traceback and cause below are verbatim from the run.

Building a spec model that groups over a string dimension crashes under
pandas 3.0. The grouped operators are affected: sum(by=…), shift(…, by=…),
sum_back(…, by=…) and at.

pandas 3.0 defaults string data to the Arrow-backed string dtype
(future.infer_string = True), so a lookup whose values are strings arrives as
an ArrowStringArray. linopy/spec/operators.py::_grouped then calls
DataArray.sel(...) on that lookup, and the installed xarray refuses to index
an ArrowStringArray (as_indexable raises TypeError: Invalid array type).

Environment

  • linopy: spec-builder branch
  • pandas 3.0.5, xarray 2026.7.0, numpy 2.4.6, Python 3.12.3

Minimal reproduction

import pandas as pd
import linopy

linopy.options["semantics"] = "v1"
spec = {
    "dimensions": {"t": {"dtype": "str"}, "g": {"dtype": "str"}},
    "lookups": {"grp": {"over": "t", "into": "g"}},
    "parameters": {"v": {"dims": ["t"]}},
    "variables": {
        "x": {"foreach": ["t"], "bounds": {"lower": 0, "upper": 100}},
        "y": {"foreach": ["t"], "bounds": {"lower": -100, "upper": 100}},
    },
    "constraints": {
        "fix": {"foreach": ["t"], "expression": "x == v"},
        "link": {"foreach": ["t"],
                 "expression": "y == shift(x, over=t, offset=1, edge=0, by=grp)"},
    },
    "objective": {"sense": "minimize", "expression": "sum(x)"},
}
data = {
    "t": ["t0", "t1", "t2"], "g": ["g0", "g1"],
    "grp": pd.Series(["g0", "g0", "g1"], index=["t0", "t1", "t2"]),
    "v": pd.Series([1.0, 2.0, 3.0], index=["t0", "t1", "t2"]),
}
linopy.Model.from_spec(spec, data)   # -> TypeError

Cause, confirmed

Adding pd.set_option("future.infer_string", False) before the build makes the
same model build and solve. The trigger is the pandas-3.0 string default
producing an ArrowStringArray for the lookup's values.

Suggested fix

In _grouped (linopy/spec/operators.py:263), take the lookup values off the
Arrow-backed dtype before indexing — build keys from a plain object/numpy
view of the values, or avoid .sel here — so string groups index the same way
int and object groups already do.

Full traceback
Traceback (most recent call last):
  File "repro.py", line 31, in <module>
    linopy.Model.from_spec(spec, data)
  File ".../linopy/model.py", line 510, in from_spec
    return cls(**model_kwargs).add_spec(spec, sources, retain=retain)
  File ".../linopy/model.py", line 494, in add_spec
    self._spec = attach(self, spec, sources, retain)
  File ".../linopy/spec/accessor.py", line 59, in attach
    build(model, bound)
  File ".../linopy/spec/builder.py", line 61, in build
    _constraints(ctx)
  File ".../linopy/spec/builder.py", line 134, in _constraints
    lhs, rhs = evaluate(row.lhs, ctx), evaluate(row.rhs, ctx)
  File ".../linopy/spec/builder.py", line 225, in evaluate
    return operators.shift(
  File ".../linopy/spec/operators.py", line 128, in shift
    groups = _grouped(over, np.asarray(array.indexes[over]), by)
  File ".../linopy/spec/operators.py", line 263, in _grouped
    keys = np.asarray(groups.sel({over: labels}).values, dtype=object)
  File ".../xarray/core/dataarray.py", line 1729, in sel
    ds = self._to_temp_dataset().sel(...)
  File ".../xarray/core/dataset.py", line 3086, in sel
    result = self.isel(indexers=query_results.dim_indexers, drop=drop)
  File ".../xarray/core/dataset.py", line 2939, in isel
    var = var.isel(var_indexers)
  File ".../xarray/core/variable.py", line 1144, in isel
    return self[key]
  File ".../xarray/core/variable.py", line 829, in __getitem__
    indexable = as_indexable(self._data)
  File ".../xarray/core/indexing.py", line 1030, in as_indexable
    raise TypeError(f"Invalid array type: {type(array)}")
TypeError: Invalid array type: <class 'pandas.arrays.ArrowStringArray'>

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions