Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ installed, `1.0-alpha`, and `1.0-alpha2`), so a single
notes in this file describe `default_version` as pinned at an earlier version, each
true until the next version shipped.

## [Unreleased]
## [1.0-alpha3] - 2026-09-02

### Added

Expand Down Expand Up @@ -224,12 +224,15 @@ true until the next version shipped.
now raise `22023`.

Retiring a group leaves its old row numbers in the visibility map, so an
index-only scan answers from the index for a row group that is gone. `expire`
cleared them; `pgcolumnar.recluster()` and the partial-group rewrite behind
`pgcolumnar.compact_rewrite()` did not, and both renumber live rows through
the same retire. All three clear now, and as of #878 all three are held by
tests; before it, only expire's clear was. The rule is that visibility-map bits go
wherever row numbers are reassigned, not only where rows expire.
index-only scan answers from the index for a row group that is gone. **None of
the three paths cleared them.** `expire`'s absence is the one that was
reported; `pgcolumnar.recluster()` and the partial-group rewrite behind
`pgcolumnar.compact_rewrite()` renumber live rows through the same retire and
had the same gap, with no report against either.

All three clear now. As of #878 all three are held by tests; before it, only
expire's clear was. The rule is that visibility-map bits go wherever row
numbers are reassigned, not only where rows expire.

`docs/sql-reference.md` gains the accepted range for `ttl_interval` and the
`NULL` rule, which is stronger than the straddling behaviour the page already
Expand Down
163 changes: 163 additions & 0 deletions RELEASE_NOTES_1.0-alpha3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
# pgColumnar 1.0-alpha3 release notes

Release date: 2026-09-02
Previous release: 1.0-alpha2 (2026-08-18)

pgColumnar is a columnar table access method for PostgreSQL. This is the third
alpha. Its theme is retention and skipping. Rows can now expire on a declared
interval. A bulk load can refuse work it has already done. Three more predicate
shapes prune whole chunk groups. The on-disk native format, PGCN v1, is
unchanged. Existing tables are read and written as before.

This release requires one upgrade command. See "Upgrading" at the end.

## Highlights

- **Retention.** `pgcolumnar.expire` drops row groups whose rows are all older
than an interval you declare on the table. It works on whole row groups, so it
reclaims space without rewriting live data.
- **Bulk loads can refuse a repeat.** `pgcolumnar.parallel_copy` records a
fingerprint of each load. A load it has already taken is refused rather than
duplicated.
- **Three more predicate shapes prune chunk groups.**
`date_trunc(unit, ts)` now drives skipping in both its range and its equality
form, and so do `IN (...)` and `= ANY(array)`.
- **The scan tells the planner what order it is in.** A sorted rewrite leaves an
ordering behind. The scan now reports it, so the planner can skip a sort.
- **The vectorized aggregate takes a wider range of queries**, including a
target list that itself contains aggregates.

## Retention

`pgcolumnar.set_options` takes `ttl_column` and `ttl_interval`.
`pgcolumnar.expire` then retires every row group whose maximum value in that
column is older than the cutoff. A group is dropped whole. Rows inside the
retention window are never touched.

The interval must be positive. A negative interval would put the cutoff in the
future, which would retire groups whose rows are still current.

## Bulk ingest

`pgcolumnar.parallel_copy` records a fingerprint for each completed load in
`pgcolumnar.load_fingerprint`. Re-running the same load is refused. This makes a
retried ingest safe to repeat after a failure, without a manual check for
partial work.

## Skipping and the planner

A predicate on `date_trunc(unit, ts)` now prunes chunk groups. Both the range
form and the equality form work. `IN (...)` and `= ANY(array)` prune too.

Skip predicates are evaluated most-selective-first, so a group that can be ruled
out cheaply is ruled out first.

The cost model and the zone-map sample now read the row-group geometry a table
was **written** with, rather than the current setting. Changing a setting no
longer reprices existing data.

## Maintenance and reporting

`pgcolumnar.vacuum_sorted()` self-gates. When the relation is already sorted on
the requested key, it does nothing rather than rewriting the table.

`pgcolumnar.sort_status` reports `sorted_kind`, so a reader can tell which kind
of ordering a table carries.

## Correctness fixes

Two defects in this list were silent: the operation looked correct and the data
was wrong. Both were found in the review before this release and both were
reproduced before being changed.

- **A projection created mid-transaction missed every write that followed it**
(#875). A write before `pgcolumnar.add_projection()` in the same transaction
left the new projection empty of everything written after it. Measured: 116
rows in the base table, 105 in the projection, with no error raised. A
covering projection scan then answered as though those rows did not exist. The
same defect in `pgcolumnar.drop_projection()` left rows in a projection
storage whose catalog rows were already deleted.
- **An Arrow import ignored the width, sign and scale the file declared**
(#881). The importer decoded with the target column's parameters instead of
the file's. A `uint64` value above 2^63 was stored as a negative number. An
`int64` file read into an `int` column returned `1,0,2,0` for `1,2,3,4`. A
`decimal(10,2)` value of 1.25 was stored as 0.0125. A
`fixed_size_binary(32)` was read as its first 16 bytes. All four imported
without an error and the wrong values were persisted. They are refused now.

The `1,0,2,0` is worth recognising if you imported integers. The reader took
its stride from the target column, so four-byte reads walked an
eight-byte-per-value buffer. Every second read landed on the high half of a
small positive number, which is zero. The signature is a real value alternating
with a zero, not a column of ascending garbage.
- Index entries for live rows are no longer destroyed (#838).
- A scrollable cursor no longer answers a backward fetch with forward rows
(#842).
- `sum(bigint)` and `avg(bigint)` no longer accumulate across a rescan (#840).
- `date_trunc(unit, ts) = 'infinity'` returns the matching row again (#836).
- An encoded NUL no longer defeats the Iceberg traversal guard (#844).
- `pgcolumnar.set_options` no longer writes past three stack arrays.
- Deleted rows no longer count toward the planner's row estimate.
- A custom scan no longer hides the children of an `INHERITS` parent.
- `TRUNCATE` retires the old storage's catalog rows, and a transaction that
writes, truncates and writes again keeps the rows it should.
- `ALTER TABLE ... SET ACCESS METHOD heap` drops the catalogs keyed to the
relation.
- A parallel export refuses a destination too long to hold the names it
generates.
- `pgcolumnar.compact_rewrite` and `pgcolumnar.maintenance_due` reject `NaN`
thresholds.

## Known issues

- **A rewrite makes a projection read as absent** (#876). `TRUNCATE`, vacuum and
recluster mint a new storage id, and the projection rows keep the old one. The
projection is still declared and its data is intact.
`pgcolumnar.rebuild_projections()` re-records them. The error message names
that function.
- **Two visibility-map clears have tests but no verdict** (#877). The clears on
the recluster and partial-rewrite paths gained coverage in this release.
Whether a defect sits behind them is not known, and there is no reported
symptom.

The rule they implement is not speculative. Its third application, on
`pgcolumnar.expire`, was a real defect: an index-only scan answered from the
index for a row group that had been retired. That one is fixed. These two are
the same rule on two other paths, with no demonstrated symptom on either.

Of the two issues in this section, **#876 can affect you today** and has a
recovery command. #877 has no known user-visible effect and is listed so the
state is on the record, not because there is something to act on.

## Upgrading

Install this build, then run the following in every database that has the
extension:

```sql
ALTER EXTENSION pgcolumnar UPDATE;
```

This is required. The upgrade adds two columns to `pgcolumnar.options` for
retention and creates the `pgcolumnar.load_fingerprint` table. It creates
`pgcolumnar.expire(regclass)`, which is the entry point for the retention
feature above. It replaces four more function definitions:
`pgcolumnar.parallel_copy` and `pgcolumnar.set_options` are dropped and
recreated at a new signature, and `pgcolumnar.maintenance_due` and
`pgcolumnar.sort_status` change in place. No table data is converted and no SQL
you write changes.

See `docs/installation.md` for the commands, including how to list the databases
that need the update.

## Scope and limitations

- This is an alpha. Interfaces may change before 1.0.
- Retention drops whole row groups. A group is retired only when every row in it
is outside the retention window.
- On PGXN this release is `1.0.0-alpha.3`, while `CREATE EXTENSION` reports
`1.0-alpha3`. PGXN requires a semantic version, which needs three integer
components. The extension's own version has two. The two names refer to the
same release.

The complete, itemized list of changes is in `CHANGELOG.md`.
2 changes: 1 addition & 1 deletion design/RELEASE_PLAN_1.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ both change the bytes on disk, so both belong in an alpha.
| `v1.0-dev` | yes | the first published marker |
| `v1.0-alpha` | 2026-08-04 | native format, scan, maintenance, Arrow and Parquet |
| `v1.0-alpha2` | 2026-08-18 | object storage, Apache Iceberg, parallel bulk ingest and export |
| `1.0-alpha3` | **not tagged** | retention, `parallel_copy` deduplication, `sort_status` reporting `sorted_kind` |
| `1.0-alpha3` | 2026-09-02 | retention, `parallel_copy` deduplication, `sort_status` reporting `sorted_kind` |

The observed cadence is 14 days, from one tag to the next. The dates below keep
it. They are a cadence, not a commitment, and each is the date the tag is cut
Expand Down
Loading