import_arrow ignores the Arrow temporal unit and reads every value as microseconds
Sibling of #864. Arrow permits s, ms, us and ns for timestamp and
time64. pgcolumnar.import_arrow decodes the integers as microseconds
whatever the schema declares, so valid data in any other unit is stored as a
different, valid-looking value. No error.
Measured on main at 8b39053, PG 18.4, pyarrow 23.0.1. Every file holds
2000-01-01 00:00:00 UTC (or 12:00:00 for time64) written correctly in its
own unit:
arrow type expected stored
timestamp('s') 2000-01-01 00:00:00 1970-01-01 00:15:46.6848 WRONG
timestamp('ms') 2000-01-01 00:00:00 1970-01-11 22:58:04.8 WRONG
timestamp('us') 2000-01-01 00:00:00 2000-01-01 00:00:00 ok
timestamp('ns') 2000-01-01 00:00:00 31969-04-01 00:00:00 WRONG
time64('us') 12:00:00 12:00:00 ok
time64('ns') 12:00:00 12000:00:00 WRONG
time64('ns') is the one worth staring at: 12000:00:00 is a legal PostgreSQL
time value, so nothing downstream can tell it from data.
Why this is filed separately from the PRs in flight
#861 already fixes it, by refusing a schema whose layout it does not decode:
main every non-us unit stored wrong
#861 every non-us unit rejected
#862 timestamp('ns') still stored wrong (31969-04-01);
time64('ns') now rejected
So #861 closes this and #864 both, and ships a test for neither. #862 in
isolation improves time64('ns') — a rejection is better than corruption — while
leaving timestamp('ns') corrupting, because 31969-04-01 passes
IS_VALID_TIMESTAMP.
This was raised as an open question by jdatcmd on #862, explicitly not asserted
because he had not traced the schema-parsing side. Both halves of his suspicion
hold: the new bounds reject valid data (time64('ns') 12:00:00) and
accept wrong data (timestamp('ns') 2000-01-01).
Reproduction
import struct, pyarrow as pa, pyarrow.ipc as ipc
v = 946684800 * 10**9 # 2000-01-01 in nanoseconds
arr = pa.Array.from_buffers(pa.timestamp('ns'), 1,
[None, pa.py_buffer(struct.pack('<q', v))])
t = pa.table({'v': arr})
with ipc.new_stream(pa.OSFile('/tmp/ns.arrows','wb'), t.schema) as w:
w.write_table(t)
CREATE TABLE nst (v timestamp) USING pgcolumnar;
SELECT pgcolumnar.import_arrow('nst', '/tmp/ns.arrows');
SELECT v FROM nst; -- 31969-04-01 00:00:00
What this asks for
Either scale by the declared unit before converting, or refuse the units that are
not decoded. Refusing is what #861 does today and it is the safe default;
scaling is the better end state, since timestamp('ns') is what Arrow producers
emit by default in several ecosystems.
Whichever is chosen, it wants a test per unit rather than per type — the three
temporal arms on #862 all use us, which is the one unit that works.
import_arrowignores the Arrow temporal unit and reads every value as microsecondsSibling of #864. Arrow permits
s,ms,usandnsfortimestampandtime64.pgcolumnar.import_arrowdecodes the integers as microsecondswhatever the schema declares, so valid data in any other unit is stored as a
different, valid-looking value. No error.
Measured on
mainat8b39053, PG 18.4, pyarrow 23.0.1. Every file holds2000-01-01 00:00:00 UTC(or12:00:00fortime64) written correctly in itsown unit:
time64('ns')is the one worth staring at:12000:00:00is a legal PostgreSQLtimevalue, so nothing downstream can tell it from data.Why this is filed separately from the PRs in flight
#861 already fixes it, by refusing a schema whose layout it does not decode:
So #861 closes this and #864 both, and ships a test for neither. #862 in
isolation improves
time64('ns')— a rejection is better than corruption — whileleaving
timestamp('ns')corrupting, because31969-04-01passesIS_VALID_TIMESTAMP.This was raised as an open question by jdatcmd on #862, explicitly not asserted
because he had not traced the schema-parsing side. Both halves of his suspicion
hold: the new bounds reject valid data (
time64('ns')12:00:00) andaccept wrong data (
timestamp('ns')2000-01-01).Reproduction
What this asks for
Either scale by the declared unit before converting, or refuse the units that are
not decoded. Refusing is what #861 does today and it is the safe default;
scaling is the better end state, since
timestamp('ns')is what Arrow producersemit by default in several ecosystems.
Whichever is chosen, it wants a test per unit rather than per type — the three
temporal arms on #862 all use
us, which is the one unit that works.