Skip to content

A valid Arrow date64 column imports as a wrong date, silently (2000-01-01 becomes 4908285-05-04) #864

Description

@OffgridwithJD

A valid Arrow date64 column imports as a wrong date, silently

pgcolumnar.import_arrow accepts an Arrow date64 column into a PostgreSQL
date column and stores a value that is not the one in the file. No error, no
warning.

Measured on main at 8b39053, PG 18.4, pyarrow 23.0.1:

date64 value (ms since epoch)   stored          should be
    946684800000                4908285-05-04   2000-01-01
              0                 1970-01-01      1970-01-01   (correct by luck)
       86400001                 238525-03-03    1970-01-02
   -1000000000000000000         4072193-07-19   (out of range, should error)
   INT64_MIN                    1970-01-01      (out of range, should error)

The second row is the reason this is filed as a bug rather than a hardening
request: 2000-01-01 is not an edge case. Any ordinary date64 file
imports garbage.

date32 is handled correctly — the same out-of-range probe is rejected:

date32 INT32_MAX  ->  rejected

so this is specific to the 8-byte date64 carrier.

Why it matters more than an out-of-range value

An out-of-range value that errors is a nuisance. A valid value that imports as a
different valid value is silent corruption: nothing in the result set says the
import went wrong, and the wrong dates are indistinguishable from data until
someone reads them.

date64 is not exotic. pyarrow produces it directly (pa.date64()), and it is
what several Arrow producers emit for date columns.

Reproduction

import struct, pyarrow as pa, pyarrow.ipc as ipc
arr = pa.Array.from_buffers(pa.date64(), 1,
        [None, pa.py_buffer(struct.pack('<q', 946684800000))])   # 2000-01-01
t = pa.table({'v': arr})
with ipc.new_stream(pa.OSFile('/tmp/d64.arrows','wb'), t.schema) as w:
    w.write_table(t)
CREATE TABLE d64t (v date) USING pgcolumnar;
SELECT pgcolumnar.import_arrow('d64t', '/tmp/d64.arrows');
SELECT v FROM d64t;     -- 4908285-05-04

Interaction with the open PRs, measured

main    ACCEPTED, stores 4908285-05-04
#861    rejected            (schema validation refuses the layout mismatch)
#862    ACCEPTED, stores 4908285-05-04

#861 already fixes this as a side effect and ships no test for it, which is a
shame, because it is a far stronger argument for schema validation than the arm
it does ship: without the Schema check, an 8-byte carrier is decoded through a
4-byte path and the value silently changes.

#862 does not address it. Its stated subject is rejecting out-of-range Arrow
temporal values, and date64 out-of-range values are still accepted
(INT64_MIN stores 1970-01-01).

What this asks for

Either a correct date64 decode — divide by 86,400,000, then apply the same
IS_VALID_DATE guard date32 gets — or an explicit refusal of the type. Both
are defensible; silently decoding it through the date32 path is not.

If #861 lands first, the layout check refuses date64 and this stops being a
correctness bug and becomes a supported-types question. That is a reason to
sequence #861 before #862 rather than the other way round.

Found while adversarially reviewing #861 and #862.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions