Skip to content

import_arrow ignores the Arrow temporal unit: a valid timestamp('ns') stores as 31969-04-01 #865

Description

@OffgridwithJD

import_arrow ignores the Arrow temporal unit and reads every value as microseconds

Sibling of #864. Arrow permits s, ms, us and ns for timestamp and
time64. pgcolumnar.import_arrow decodes the integers as microseconds
whatever the schema declares, so valid data in any other unit is stored as a
different, valid-looking value.
No error.

Measured on main at 8b39053, PG 18.4, pyarrow 23.0.1. Every file holds
2000-01-01 00:00:00 UTC (or 12:00:00 for time64) written correctly in its
own unit:

arrow type          expected               stored
timestamp('s')      2000-01-01 00:00:00    1970-01-01 00:15:46.6848    WRONG
timestamp('ms')     2000-01-01 00:00:00    1970-01-11 22:58:04.8       WRONG
timestamp('us')     2000-01-01 00:00:00    2000-01-01 00:00:00         ok
timestamp('ns')     2000-01-01 00:00:00    31969-04-01 00:00:00        WRONG
time64('us')        12:00:00               12:00:00                    ok
time64('ns')        12:00:00               12000:00:00                 WRONG

time64('ns') is the one worth staring at: 12000:00:00 is a legal PostgreSQL
time value, so nothing downstream can tell it from data.

Why this is filed separately from the PRs in flight

#861 already fixes it, by refusing a schema whose layout it does not decode:

main    every non-us unit stored wrong
#861    every non-us unit rejected
#862    timestamp('ns') still stored wrong (31969-04-01);
        time64('ns') now rejected

So #861 closes this and #864 both, and ships a test for neither. #862 in
isolation improves time64('ns') — a rejection is better than corruption — while
leaving timestamp('ns') corrupting, because 31969-04-01 passes
IS_VALID_TIMESTAMP.

This was raised as an open question by jdatcmd on #862, explicitly not asserted
because he had not traced the schema-parsing side. Both halves of his suspicion
hold: the new bounds reject valid data (time64('ns') 12:00:00) and
accept wrong data (timestamp('ns') 2000-01-01).

Reproduction

import struct, pyarrow as pa, pyarrow.ipc as ipc
v = 946684800 * 10**9                      # 2000-01-01 in nanoseconds
arr = pa.Array.from_buffers(pa.timestamp('ns'), 1,
        [None, pa.py_buffer(struct.pack('<q', v))])
t = pa.table({'v': arr})
with ipc.new_stream(pa.OSFile('/tmp/ns.arrows','wb'), t.schema) as w:
    w.write_table(t)
CREATE TABLE nst (v timestamp) USING pgcolumnar;
SELECT pgcolumnar.import_arrow('nst', '/tmp/ns.arrows');
SELECT v FROM nst;     -- 31969-04-01 00:00:00

What this asks for

Either scale by the declared unit before converting, or refuse the units that are
not decoded. Refusing is what #861 does today and it is the safe default;
scaling is the better end state, since timestamp('ns') is what Arrow producers
emit by default in several ecosystems.

Whichever is chosen, it wants a test per unit rather than per type — the three
temporal arms on #862 all use us, which is the one unit that works.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions