DATE_TRUNC is evaluated entirely by the query engine today. It appears in ClickBench Q42 and SpatialBench Q3/Q5. Vortex should support it so engines can push it into the scan and prune zones with it.
Approach: function plus comparison rewrite
date_trunc(u, ts) is a step function that never decreases, so comparing it with a constant is exactly a range check on ts. The bounds are computed once, at planning time:
date_trunc(u, ts) = X -> ts >= X AND ts < next(X) (X aligned to u)
-> false, null for null ts (X not aligned)
date_trunc(u, ts) < X -> ts < ceil(X)
date_trunc(u, ts) <= X -> ts < next(floor(X))
date_trunc(u, ts) >= X -> ts >= ceil(X)
date_trunc(u, ts) > X -> ts >= next(floor(X))
IN / <> -> OR of ranges / negated range
- Filters against a constant use the rewrite. Pruning comes for free from the existing min/max rules. Comparisons are cheaper than truncation and can run on compressed data. Zoned timestamps and
DATE inputs are handled exactly, because the bounds are computed on a single value.
- Everything else uses the function. That covers projections, group-by keys, non-constant comparisons such as
date_trunc(a) = date_trunc(b), and date_trunc nested inside other expressions.
1. Missing function
Vortex has no date_trunc scalar function. Add vortex.date_trunc:
- Input is a
vortex.timestamp that is naive or UTC. The output has the same dtype.
- Units run from microsecond to year, with weeks starting on Monday.
- The kernel is integer arithmetic in the storage unit, with no per-row calendar conversion.
2. Semantics in DataFusion and DuckDB
|
DataFusion |
DuckDB |
Proposed Vortex |
| Inputs |
Timestamp, Time. Dates are cast to timestamp first. |
TIMESTAMP, DATE, INTERVAL. TIMESTAMPTZ goes through ICU. |
Naive or UTC timestamp |
| Returns |
The input type |
TIMESTAMP, or INTERVAL for interval input |
The input type |
| Extra units |
none |
millennium, century, decade, isoyear |
none |
| Time zones |
Truncates on the local calendar |
Truncates in the session time zone |
UTC only in the function; any zone in the rewrite |
| Stats |
preimage exists for date_part, not for date_trunc |
Propagates min/max |
See section 4 |
On the inputs Vortex accepts, the results match both engines.
3. Filter pushdown
4. Pruning
Once rewritten, comparisons against a constant prune through the existing rules. The function itself cannot prune yet, because the zone-map binder only has stats for bare columns.
Reference
Prototype branch, which needs refining: https://github.com/vortex-data/vortex/tree/ji/nifty-mccarthy-7fw1jr
It has the function, a divan benchmark and DataFusion pushdown. It has no rewrite, no DuckDB support and no pruning. In early numbers, the kernel is about 1.1x to 1.8x faster than DataFusion's, and a date_trunc('day', ...) filter runs about 2x faster.
DATE_TRUNCis evaluated entirely by the query engine today. It appears in ClickBench Q42 and SpatialBench Q3/Q5. Vortex should support it so engines can push it into the scan and prune zones with it.Approach: function plus comparison rewrite
date_trunc(u, ts)is a step function that never decreases, so comparing it with a constant is exactly a range check onts. The bounds are computed once, at planning time:DATEinputs are handled exactly, because the bounds are computed on a single value.date_trunc(a) = date_trunc(b), anddate_truncnested inside other expressions.1. Missing function
Vortex has no
date_truncscalar function. Addvortex.date_trunc:vortex.timestampthat is naive or UTC. The output has the same dtype.2. Semantics in DataFusion and DuckDB
preimageexists fordate_part, not fordate_truncOn the inputs Vortex accepts, the results match both engines.
3. Filter pushdown
date_trunc(x)and a constant into the ranges above. The "not aligned" case must stay null for null inputs so thatNOT (...)remains correct.date_trunc(<literal>, <timestamp>)in predicates and projections. Also propose adate_truncpreimageupstream, which would help Parquet too.date_truncinvortex-duckdbforTIMESTAMPinput.DATEinput to the function. Accept dates, or add a date-to-timestamp cast.date_truncrather than filter on it. DataFusion evaluates the group key inside the aggregate, so it never reaches the scan. Pushing it down needs an optimizer rule that lifts the key into a projection below the aggregate.4. Pruning
Once rewritten, comparisons against a constant prune through the existing rules. The function itself cannot prune yet, because the zone-map binder only has stats for bare columns.
min/maxofdate_trunc(u, x)todate_trunc(u, min/max(x)), alongside the existingCastcase instats/rewrite/builtins.rs. This is exact becausedate_truncnever decreases as its input increases, and it is what DuckDB does.date_truncover time-sorted data.Reference
Prototype branch, which needs refining: https://github.com/vortex-data/vortex/tree/ji/nifty-mccarthy-7fw1jr
It has the function, a divan benchmark and DataFusion pushdown. It has no rewrite, no DuckDB support and no pruning. In early numbers, the kernel is about 1.1x to 1.8x faster than DataFusion's, and a
date_trunc('day', ...)filter runs about 2x faster.