Make arrays of several integer types adaptable

[roed-math]

The bug

psycopg picks one dumper for a whole array, and refuses a list whose entries have different types unless those types dump to the same Postgres oid. SageIntegerDumper has oid 0 (unknown, so the server infers the type from context, deliberately mimicking what psycopg2's client-side interpolation did) while Python int dumps as int2/int4/int8/numeric by magnitude, so the two never agree:

db.nf_fields.lucky({'coeffs': [ZZ(1), -1, 1]})
# psycopg.DataError: cannot dump lists of mixed types; got: Integer, int

Homogeneous lists are fine either way, so this only bites a caller that assembles an array out of more than one source: a computed value here, an int literal there. Under psycopg2 such a list worked, so the driver port turned every one of those call sites into a 500 without any of them changing.

Two are known in LMFDB. The CM field lookup on elliptic curve isogeny class pages was found in production and patched at the call site in LMFDB#7129. The L-function Euler product search is the second, found while writing this and confirmed against the LMFDB mirror: parse_euler pads a Sage coefficient list with Python int zeros, so any entered Euler factor of less than the searched degree (F2=1-T at degree 4, say) fails. Neither is a psycodict bug exactly, but the failure mode is invisible until the untested branch runs, which makes it worth a net rather than another round of call-site fixes.

The fix

homogenize_int_arrays in encoding.py, applied to a statement's values from _execute — the one place every statement psycodict runs passes through, so searches, counts, statistics, inserts and updates are all covered by the single call.

It is deliberately narrow: a list is only rewritten when it would otherwise have raised. There must be at least two distinct element types, every one of them integral, none of them a bool (which Python counts as an integer and Postgres does not), and at least one of them not an int subclass. That last condition leaves psycopg's own Int2/Int4/Int8/IntNumeric wrappers alone: they subclass int precisely to pin down an oid, and replacing them with plain ints would throw away the choice. So no query that works today changes behaviour, and SageIntegerDumper keeps its oid 0 — giving it a real oid would have been the other way to make the types agree, but it would break a Sage Integer compared against a text column, which oid 0 handles today.

A nested array is flattened first, since Postgres and psycopg both treat it as a single array whose entries must agree throughout rather than row by row.

When nothing needs fixing the values come back as the same object, so ordinary queries copy nothing.

Two things only came out of testing:

• insert_many binds named parameters (%(col)s) with dict rows, so the values walker has to handle mappings and not just sequences. The first version silently did nothing there.
• quick_count serializes the query itself to json for the statistics cache, so a value type accepted in a query has to be encodable there too. Json.prep gets the matching numbers.Integral fallback, placed after the (str, bool, float, int) branch so that a bool stays a bool.

Tests

tests/conftest.py grows a FakeInteger registered under numbers.Integral — psycodict does not depend on Sage, so the type this exists for cannot be imported by the unit jobs, and registering reproduces everything the code and psycopg look at. Covered: the conversions, everything deliberately left alone, each of psycopg's oid wrappers, both values shapes, the no-copy property, and a Sage-mode test using the real Integer on the shape that broke the isogeny class pages. Round-trip tests against a real database go through flat integer[] and numeric[] columns, a two-dimensional column, the array operators, insert_many and update.

Full suite green under Sage and without it, ruff clean. Verified by hand against the LMFDB mirror: the CM field lookup, the Euler product search and a nested integer[] column all fail on 1.0.0rc2 and pass on this branch.

[read-the-docs-community]

Documentation build overview

📚 psycodict | 🛠️ Build #33919630 | 📁 Comparing a0670eb against latest (a161f5b)

  🔍 Preview build  

4 files changed

± genindex.html
± api/encoding.html
± _modules/psycodict/base.html
± _modules/psycodict/encoding.html