Skip to content

Add date_trunc #10023

Description

@joseph-isaacs

DATE_TRUNC is evaluated entirely by the query engine today. It appears in ClickBench Q42 and SpatialBench Q3/Q5. Vortex should support it so engines can push it into the scan and prune zones with it.

Approach: function plus comparison rewrite

date_trunc(u, ts) is a step function that never decreases, so comparing it with a constant is exactly a range check on ts. The bounds are computed once, at planning time:

date_trunc(u, ts) =  X   ->  ts >= X AND ts < next(X)     (X aligned to u)
                          ->  false, null for null ts      (X not aligned)
date_trunc(u, ts) <  X   ->  ts <  ceil(X)
date_trunc(u, ts) <= X   ->  ts <  next(floor(X))
date_trunc(u, ts) >= X   ->  ts >= ceil(X)
date_trunc(u, ts) >  X   ->  ts >= next(floor(X))
IN / <>                  ->  OR of ranges / negated range
  • Filters against a constant use the rewrite. Pruning comes for free from the existing min/max rules. Comparisons are cheaper than truncation and can run on compressed data. Zoned timestamps and DATE inputs are handled exactly, because the bounds are computed on a single value.
  • Everything else uses the function. That covers projections, group-by keys, non-constant comparisons such as date_trunc(a) = date_trunc(b), and date_trunc nested inside other expressions.

1. Missing function

Vortex has no date_trunc scalar function. Add vortex.date_trunc:

  • Input is a vortex.timestamp that is naive or UTC. The output has the same dtype.
  • Units run from microsecond to year, with weeks starting on Monday.
  • The kernel is integer arithmetic in the storage unit, with no per-row calendar conversion.

2. Semantics in DataFusion and DuckDB

DataFusion DuckDB Proposed Vortex
Inputs Timestamp, Time. Dates are cast to timestamp first. TIMESTAMP, DATE, INTERVAL. TIMESTAMPTZ goes through ICU. Naive or UTC timestamp
Returns The input type TIMESTAMP, or INTERVAL for interval input The input type
Extra units none millennium, century, decade, isoyear none
Time zones Truncates on the local calendar Truncates in the session time zone UTC only in the function; any zone in the rewrite
Stats preimage exists for date_part, not for date_trunc Propagates min/max See section 4

On the inputs Vortex accepts, the results match both engines.

3. Filter pushdown

  • Comparison rewrite. Add a Vortex simplify rule that turns a comparison between date_trunc(x) and a constant into the ranges above. The "not aligned" case must stay null for null inputs so that NOT (...) remains correct.
  • DataFusion. Convert date_trunc(<literal>, <timestamp>) in predicates and projections. Also propose a date_trunc preimage upstream, which would help Parquet too.
  • DuckDB. Map date_trunc in vortex-duckdb for TIMESTAMP input.
  • DATE input to the function. Accept dates, or add a date-to-timestamp cast.
  • Group-by keys (related). The benchmark queries group by date_trunc rather than filter on it. DataFusion evaluates the group key inside the aggregate, so it never reaches the scan. Pushing it down needs an optimizer rule that lifts the key into a projection below the aggregate.

4. Pruning

Once rewritten, comparisons against a constant prune through the existing rules. The function itself cannot prune yet, because the zone-map binder only has stats for bare columns.

  • Min/max propagation for the non-constant cases. Map min/max of date_trunc(u, x) to date_trunc(u, min/max(x)), alongside the existing Cast case in stats/rewrite/builtins.rs. This is exact because date_trunc never decreases as its input increases, and it is what DuckDB does.
  • Benchmark. Add a query that filters on date_trunc over time-sorted data.

Reference

Prototype branch, which needs refining: https://github.com/vortex-data/vortex/tree/ji/nifty-mccarthy-7fw1jr

It has the function, a divan benchmark and DataFusion pushdown. It has no rewrite, no DuckDB support and no pruning. In early numbers, the kernel is about 1.1x to 1.8x faster than DataFusion's, and a date_trunc('day', ...) filter runs about 2x faster.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions