Add TimestampWithOffset canonical extension type
#8743
+546
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
westonpace
approved these changes
Nov 4, 2025
Member
westonpace
left a comment
westonpace
left a comment
There was a problem hiding this comment.
This seems pretty straightforward and reasonable
alamb
changed the title
TimestampWithOffset extension typeTimestampWithOffset extension type
Contributor
|
There are several TODOs in this PRs description. Is that intended? |
Author
|
@alamb We're currently still discussing the FORMAT in the For
Realistically, even after the FORMAT PR goes into the spec I think this will take me a few more weeks to fully flesh out. |
Contributor
|
Thanks for the clarification @serramatutu -- I'll mark it as a draft again then |
felipecrv
added a commit
to apache/arrow
that referenced
this pull request
Dec 5, 2025
…48002) ### Rationale for this change Closes #44248 Arrow has no built-in canonical way of representing the `TIMESTAMP WITH TIME ZONE` SQL type, which is present across multiple different database systems. Not having a native way to represent this forces users to either convert to UTC and drop the time zone, which may have correctness implications, or use bespoke workarounds. A new `arrow.timestamp_with_offset` extension type would introduce a standard canonical way of representing that information. Rust implementation: apache/arrow-rs#8743 Go implementation: apache/arrow-go#558 [DISCUSS] [thread in the mailing list](https://lists.apache.org/thread/yhbr3rj9l59yoxv92o2s6dqlop16sfnk). ### What changes are included in this PR? Proposal and documentation for `arrow.timestamp_with_offset` canonical extension type. ### Are these changes tested? N/A ### Are there any user-facing changes? Yes, this is an extension to the arrow format. * GitHub Issue: #44248 --------- Co-authored-by: David Li <li.davidm96@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
This commit adds a new `TimestampWithOffset` extension type. This type represents a timestamp column that stores potentially different timezone offsets per value. The timestamp is stored in UTC alongside the original timezone offset in minutes.
serramatutu
force-pushed
the
serramatutu/TimestampWithOffset/rs
branch
from
d187bb9 to
f6cda58
Compare
alamb
approved these changes
Dec 18, 2025
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Looks good to me -- thank you @serramatutu and @westonpace
| /// The extension type for `TimestampWithOffset`. | ||
| /// | ||
| /// <https://arrow.apache.org/docs/format/CanonicalExtensions.html#timestamp-with-offset> | ||
| TimestampWithOffset(TimestampWithOffset), |
Contributor
There was a problem hiding this comment.
Since this is a pub enum, unfortunately this will need to wait until the next major arrow release (in Jan):
alamb
changed the title
TimestampWithOffset extension typeTimestampWithOffset canonical extension type
Author
|
Looks like we got a CI flake and need to rerun:
|
felipecrv
approved these changes
Dec 29, 2025
Comment on lines
+28
to
+51
| /// This type represents a timestamp column that stores potentially different timezone offsets per value. | ||
| /// The timestamp is stored in UTC alongside the original timezone offset in minutes. This extension type | ||
| /// is intended to be compatible with ANSI SQL's `TIMESTAMP WITH TIME ZONE`, which is supported by multiple | ||
| /// database engines. | ||
| /// | ||
| /// The storage type of the extension is a `Struct` with 2 fields, in order: | ||
| /// - `timestamp`: a non-nullable `Timestamp(time_unit, "UTC")`, where `time_unit` is any Arrow `TimeUnit` (s, ms, us or ns). | ||
| /// - `offset_minutes`: a non-nullable signed 16-bit integer (`Int16`) representing the offset in minutes | ||
| /// from the UTC timezone. Negative offsets represent time zones west of UTC, while positive offsets represent | ||
| /// east. Offsets normally range from -779 (-12:59) to +780 (+13:00). | ||
| /// | ||
| /// This type has no type parameters. | ||
| /// | ||
| /// Metadata is either empty or an empty string. | ||
| /// | ||
| /// It is also *permissible* for the `offset_minutes` field to be dictionary-encoded with a preferred (*but not required*) | ||
| /// index type of `int8`, or run-end-encoded with a preferred (*but not required*) runs type of `int8`. | ||
| /// | ||
| /// It's worth noting that the data source needs to resolve timezone strings such as `UTC` or | ||
| /// `Americas/Los_Angeles` into an offset in minutes in order to construct a `TimestampWithOffset`. | ||
| /// This makes `TimestampWithOffset` type "lossy" in the sense that any original "unresolved" | ||
| /// timezone string gets lost in this conversion. It's a tradeoff for optimizing the row | ||
| /// representation and simplifying the client code, which does not need to know how to convert | ||
| /// from timezone string to its corresponding offset in minutes. |
Contributor
There was a problem hiding this comment.
Can you gq the paragraphs on vim? Or the equivalent on your editor? Keeping them around 70 characters with uniform length.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Implement
arrow.timestamp_with_offsetcanonical extension type.Rationale for this change
Be compliant with Arrow spec: apache/arrow#48002
What changes are included in this PR?
This commit adds a new
TimestampWithOffsetextension type. This type represents a timestamp column that stores potentially different timezone offsets per value. The timestamp is stored in UTC alongside the original timezone offset in minutes.Are these changes tested?
Yes.
Are there any user-facing changes?
Yes, this is a new canonical extension type.