A valid Arrow date64 column imports as a wrong date, silently
pgcolumnar.import_arrow accepts an Arrow date64 column into a PostgreSQL
date column and stores a value that is not the one in the file. No error, no
warning.
Measured on main at 8b39053, PG 18.4, pyarrow 23.0.1:
date64 value (ms since epoch) stored should be
946684800000 4908285-05-04 2000-01-01
0 1970-01-01 1970-01-01 (correct by luck)
86400001 238525-03-03 1970-01-02
-1000000000000000000 4072193-07-19 (out of range, should error)
INT64_MIN 1970-01-01 (out of range, should error)
The second row is the reason this is filed as a bug rather than a hardening
request: 2000-01-01 is not an edge case. Any ordinary date64 file
imports garbage.
date32 is handled correctly — the same out-of-range probe is rejected:
date32 INT32_MAX -> rejected
so this is specific to the 8-byte date64 carrier.
Why it matters more than an out-of-range value
An out-of-range value that errors is a nuisance. A valid value that imports as a
different valid value is silent corruption: nothing in the result set says the
import went wrong, and the wrong dates are indistinguishable from data until
someone reads them.
date64 is not exotic. pyarrow produces it directly (pa.date64()), and it is
what several Arrow producers emit for date columns.
Reproduction
import struct, pyarrow as pa, pyarrow.ipc as ipc
arr = pa.Array.from_buffers(pa.date64(), 1,
[None, pa.py_buffer(struct.pack('<q', 946684800000))]) # 2000-01-01
t = pa.table({'v': arr})
with ipc.new_stream(pa.OSFile('/tmp/d64.arrows','wb'), t.schema) as w:
w.write_table(t)
CREATE TABLE d64t (v date) USING pgcolumnar;
SELECT pgcolumnar.import_arrow('d64t', '/tmp/d64.arrows');
SELECT v FROM d64t; -- 4908285-05-04
Interaction with the open PRs, measured
main ACCEPTED, stores 4908285-05-04
#861 rejected (schema validation refuses the layout mismatch)
#862 ACCEPTED, stores 4908285-05-04
#861 already fixes this as a side effect and ships no test for it, which is a
shame, because it is a far stronger argument for schema validation than the arm
it does ship: without the Schema check, an 8-byte carrier is decoded through a
4-byte path and the value silently changes.
#862 does not address it. Its stated subject is rejecting out-of-range Arrow
temporal values, and date64 out-of-range values are still accepted
(INT64_MIN stores 1970-01-01).
What this asks for
Either a correct date64 decode — divide by 86,400,000, then apply the same
IS_VALID_DATE guard date32 gets — or an explicit refusal of the type. Both
are defensible; silently decoding it through the date32 path is not.
If #861 lands first, the layout check refuses date64 and this stops being a
correctness bug and becomes a supported-types question. That is a reason to
sequence #861 before #862 rather than the other way round.
Found while adversarially reviewing #861 and #862.
A valid Arrow
date64column imports as a wrong date, silentlypgcolumnar.import_arrowaccepts an Arrowdate64column into a PostgreSQLdatecolumn and stores a value that is not the one in the file. No error, nowarning.
Measured on
mainat8b39053, PG 18.4, pyarrow 23.0.1:The second row is the reason this is filed as a bug rather than a hardening
request:
2000-01-01is not an edge case. Any ordinarydate64fileimports garbage.
date32is handled correctly — the same out-of-range probe is rejected:so this is specific to the 8-byte
date64carrier.Why it matters more than an out-of-range value
An out-of-range value that errors is a nuisance. A valid value that imports as a
different valid value is silent corruption: nothing in the result set says the
import went wrong, and the wrong dates are indistinguishable from data until
someone reads them.
date64is not exotic.pyarrowproduces it directly (pa.date64()), and it iswhat several Arrow producers emit for date columns.
Reproduction
Interaction with the open PRs, measured
#861 already fixes this as a side effect and ships no test for it, which is a
shame, because it is a far stronger argument for schema validation than the arm
it does ship: without the Schema check, an 8-byte carrier is decoded through a
4-byte path and the value silently changes.
#862 does not address it. Its stated subject is rejecting out-of-range Arrow
temporal values, and
date64out-of-range values are still accepted(
INT64_MINstores1970-01-01).What this asks for
Either a correct
date64decode — divide by 86,400,000, then apply the sameIS_VALID_DATEguarddate32gets — or an explicit refusal of the type. Bothare defensible; silently decoding it through the
date32path is not.If #861 lands first, the layout check refuses
date64and this stops being acorrectness bug and becomes a supported-types question. That is a reason to
sequence #861 before #862 rather than the other way round.
Found while adversarially reviewing #861 and #862.