From e929c26a3a4780f807c4e5a2df44dc46c993ff16 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 2 Sep 2026 20:20:15 -0600 Subject: [PATCH] docs: release notes for 1.0-alpha3, and close the changelog for it Three files were still describing an unreleased version after the tag was cut. RELEASE_NOTES_1.0-alpha3.md, following the shape of the alpha2 notes: highlights, then a section per theme, then correctness fixes, known issues, upgrading and scope. Two entries under correctness say plainly that the defect was SILENT and give the measured numbers, because that is what a reader needs to decide whether they were affected. ONE OF THOSE NUMBERS WAS WRONG AND IS THE REASON TO READ THIS MESSAGE. I first published #881's int64-into-int case as "returned 0,0,1,2 for 1,2,3,4". It returns 1,0,2,0. My probe was string_agg(c0::text, ',' ORDER BY c0) so I sorted the multiset and published the sort as the read-back order. The reviewer caught it, and reports having made the identical mistake first with an unordered string_agg, catching it only because the two multisets matched. It matters past pedantry. 0,0,1,2 implies a positional mapping and sends a user looking for a column of ascending garbage. The mechanism gives something far more recognisable: the stride comes from the target column, so four-byte reads walk an eight-byte-per-value buffer and every second read lands on the high half of a small positive number, which is zero. A real value alternating with a zero. That is the signature now published, with the explanation. Verified per row rather than by aggregate, on a pre-fix build with the guard absent: ctid (0,2) -> 1 (0,3) -> 0 (0,4) -> 2 (0,5) -> 0 The other three numbers are exact and were independently confirmed: 116 base rows against 105 in the projection; uint64 2^63+5 stored as -9223372036854775803; decimal(10,2) 1.25 stored as 0.0125. UPGRADING: the reviewer RAN the upgrade rather than reading it and all four claims hold. It also creates pgcolumnar.expire(regclass), the entry point for the headline retention feature, which the section did not mention -- a reader learned a fingerprint table appears but not that the feature's own function is new. Added, along with which functions are dropped-and-recreated at a new signature and which change in place. KNOWN ISSUES ARE LISTED RATHER THAN OMITTED, and now say which one to act on. #876 can affect a reader today and has a recovery command. #877 has no known user-visible symptom. Sitting side by side they read as comparable exposure and they are not. The #877 entry also records that the same rule's THIRD application, on expire, WAS a real defect, so the rule is not speculative -- these two are the same rule on paths with no demonstrated symptom. CHANGELOG.md: [Unreleased] becomes [1.0-alpha3] - 2026-09-02, 81 entries. And one sentence in it was FALSE, found by checking the release notes against it: the #403 entry said "expire cleared them; recluster and the partial-group rewrite did not", which reads as expire having cleared the visibility map all along. Measured at 381c765, the merge-base: src/columnar_vacuum.c holds NO clearing call of any kind -- not PgColumnarVMClearForRowRange, not visibilitymap_*, nothing. All three arrive together. None of the three cleared; expire is simply the one that was reported. Corrected, because after this merge the notes and the changelog ship three files apart in the same distribution. design/RELEASE_PLAN_1.0.md: the table said 1.0-alpha3 was "not tagged". Prose checked with test/plain_language_check.py, the checker docs_style runs. RELEASE_NOTES_*.md is outside that gate, so this was voluntary. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017V7PhZ1TzoVVNsACFXTbdT --- CHANGELOG.md | 17 ++-- RELEASE_NOTES_1.0-alpha3.md | 163 ++++++++++++++++++++++++++++++++++++ design/RELEASE_PLAN_1.0.md | 2 +- 3 files changed, 174 insertions(+), 8 deletions(-) create mode 100644 RELEASE_NOTES_1.0-alpha3.md diff --git a/CHANGELOG.md b/CHANGELOG.md index ea78dd23..896d687c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,7 @@ installed, `1.0-alpha`, and `1.0-alpha2`), so a single notes in this file describe `default_version` as pinned at an earlier version, each true until the next version shipped. -## [Unreleased] +## [1.0-alpha3] - 2026-09-02 ### Added @@ -224,12 +224,15 @@ true until the next version shipped. now raise `22023`. Retiring a group leaves its old row numbers in the visibility map, so an - index-only scan answers from the index for a row group that is gone. `expire` - cleared them; `pgcolumnar.recluster()` and the partial-group rewrite behind - `pgcolumnar.compact_rewrite()` did not, and both renumber live rows through - the same retire. All three clear now, and as of #878 all three are held by - tests; before it, only expire's clear was. The rule is that visibility-map bits go - wherever row numbers are reassigned, not only where rows expire. + index-only scan answers from the index for a row group that is gone. **None of + the three paths cleared them.** `expire`'s absence is the one that was + reported; `pgcolumnar.recluster()` and the partial-group rewrite behind + `pgcolumnar.compact_rewrite()` renumber live rows through the same retire and + had the same gap, with no report against either. + + All three clear now. As of #878 all three are held by tests; before it, only + expire's clear was. The rule is that visibility-map bits go wherever row + numbers are reassigned, not only where rows expire. `docs/sql-reference.md` gains the accepted range for `ttl_interval` and the `NULL` rule, which is stronger than the straddling behaviour the page already diff --git a/RELEASE_NOTES_1.0-alpha3.md b/RELEASE_NOTES_1.0-alpha3.md new file mode 100644 index 00000000..be9ff465 --- /dev/null +++ b/RELEASE_NOTES_1.0-alpha3.md @@ -0,0 +1,163 @@ +# pgColumnar 1.0-alpha3 release notes + +Release date: 2026-09-02 +Previous release: 1.0-alpha2 (2026-08-18) + +pgColumnar is a columnar table access method for PostgreSQL. This is the third +alpha. Its theme is retention and skipping. Rows can now expire on a declared +interval. A bulk load can refuse work it has already done. Three more predicate +shapes prune whole chunk groups. The on-disk native format, PGCN v1, is +unchanged. Existing tables are read and written as before. + +This release requires one upgrade command. See "Upgrading" at the end. + +## Highlights + +- **Retention.** `pgcolumnar.expire` drops row groups whose rows are all older + than an interval you declare on the table. It works on whole row groups, so it + reclaims space without rewriting live data. +- **Bulk loads can refuse a repeat.** `pgcolumnar.parallel_copy` records a + fingerprint of each load. A load it has already taken is refused rather than + duplicated. +- **Three more predicate shapes prune chunk groups.** + `date_trunc(unit, ts)` now drives skipping in both its range and its equality + form, and so do `IN (...)` and `= ANY(array)`. +- **The scan tells the planner what order it is in.** A sorted rewrite leaves an + ordering behind. The scan now reports it, so the planner can skip a sort. +- **The vectorized aggregate takes a wider range of queries**, including a + target list that itself contains aggregates. + +## Retention + +`pgcolumnar.set_options` takes `ttl_column` and `ttl_interval`. +`pgcolumnar.expire` then retires every row group whose maximum value in that +column is older than the cutoff. A group is dropped whole. Rows inside the +retention window are never touched. + +The interval must be positive. A negative interval would put the cutoff in the +future, which would retire groups whose rows are still current. + +## Bulk ingest + +`pgcolumnar.parallel_copy` records a fingerprint for each completed load in +`pgcolumnar.load_fingerprint`. Re-running the same load is refused. This makes a +retried ingest safe to repeat after a failure, without a manual check for +partial work. + +## Skipping and the planner + +A predicate on `date_trunc(unit, ts)` now prunes chunk groups. Both the range +form and the equality form work. `IN (...)` and `= ANY(array)` prune too. + +Skip predicates are evaluated most-selective-first, so a group that can be ruled +out cheaply is ruled out first. + +The cost model and the zone-map sample now read the row-group geometry a table +was **written** with, rather than the current setting. Changing a setting no +longer reprices existing data. + +## Maintenance and reporting + +`pgcolumnar.vacuum_sorted()` self-gates. When the relation is already sorted on +the requested key, it does nothing rather than rewriting the table. + +`pgcolumnar.sort_status` reports `sorted_kind`, so a reader can tell which kind +of ordering a table carries. + +## Correctness fixes + +Two defects in this list were silent: the operation looked correct and the data +was wrong. Both were found in the review before this release and both were +reproduced before being changed. + +- **A projection created mid-transaction missed every write that followed it** + (#875). A write before `pgcolumnar.add_projection()` in the same transaction + left the new projection empty of everything written after it. Measured: 116 + rows in the base table, 105 in the projection, with no error raised. A + covering projection scan then answered as though those rows did not exist. The + same defect in `pgcolumnar.drop_projection()` left rows in a projection + storage whose catalog rows were already deleted. +- **An Arrow import ignored the width, sign and scale the file declared** + (#881). The importer decoded with the target column's parameters instead of + the file's. A `uint64` value above 2^63 was stored as a negative number. An + `int64` file read into an `int` column returned `1,0,2,0` for `1,2,3,4`. A + `decimal(10,2)` value of 1.25 was stored as 0.0125. A + `fixed_size_binary(32)` was read as its first 16 bytes. All four imported + without an error and the wrong values were persisted. They are refused now. + + The `1,0,2,0` is worth recognising if you imported integers. The reader took + its stride from the target column, so four-byte reads walked an + eight-byte-per-value buffer. Every second read landed on the high half of a + small positive number, which is zero. The signature is a real value alternating + with a zero, not a column of ascending garbage. +- Index entries for live rows are no longer destroyed (#838). +- A scrollable cursor no longer answers a backward fetch with forward rows + (#842). +- `sum(bigint)` and `avg(bigint)` no longer accumulate across a rescan (#840). +- `date_trunc(unit, ts) = 'infinity'` returns the matching row again (#836). +- An encoded NUL no longer defeats the Iceberg traversal guard (#844). +- `pgcolumnar.set_options` no longer writes past three stack arrays. +- Deleted rows no longer count toward the planner's row estimate. +- A custom scan no longer hides the children of an `INHERITS` parent. +- `TRUNCATE` retires the old storage's catalog rows, and a transaction that + writes, truncates and writes again keeps the rows it should. +- `ALTER TABLE ... SET ACCESS METHOD heap` drops the catalogs keyed to the + relation. +- A parallel export refuses a destination too long to hold the names it + generates. +- `pgcolumnar.compact_rewrite` and `pgcolumnar.maintenance_due` reject `NaN` + thresholds. + +## Known issues + +- **A rewrite makes a projection read as absent** (#876). `TRUNCATE`, vacuum and + recluster mint a new storage id, and the projection rows keep the old one. The + projection is still declared and its data is intact. + `pgcolumnar.rebuild_projections()` re-records them. The error message names + that function. +- **Two visibility-map clears have tests but no verdict** (#877). The clears on + the recluster and partial-rewrite paths gained coverage in this release. + Whether a defect sits behind them is not known, and there is no reported + symptom. + + The rule they implement is not speculative. Its third application, on + `pgcolumnar.expire`, was a real defect: an index-only scan answered from the + index for a row group that had been retired. That one is fixed. These two are + the same rule on two other paths, with no demonstrated symptom on either. + + Of the two issues in this section, **#876 can affect you today** and has a + recovery command. #877 has no known user-visible effect and is listed so the + state is on the record, not because there is something to act on. + +## Upgrading + +Install this build, then run the following in every database that has the +extension: + +```sql +ALTER EXTENSION pgcolumnar UPDATE; +``` + +This is required. The upgrade adds two columns to `pgcolumnar.options` for +retention and creates the `pgcolumnar.load_fingerprint` table. It creates +`pgcolumnar.expire(regclass)`, which is the entry point for the retention +feature above. It replaces four more function definitions: +`pgcolumnar.parallel_copy` and `pgcolumnar.set_options` are dropped and +recreated at a new signature, and `pgcolumnar.maintenance_due` and +`pgcolumnar.sort_status` change in place. No table data is converted and no SQL +you write changes. + +See `docs/installation.md` for the commands, including how to list the databases +that need the update. + +## Scope and limitations + +- This is an alpha. Interfaces may change before 1.0. +- Retention drops whole row groups. A group is retired only when every row in it + is outside the retention window. +- On PGXN this release is `1.0.0-alpha.3`, while `CREATE EXTENSION` reports + `1.0-alpha3`. PGXN requires a semantic version, which needs three integer + components. The extension's own version has two. The two names refer to the + same release. + +The complete, itemized list of changes is in `CHANGELOG.md`. diff --git a/design/RELEASE_PLAN_1.0.md b/design/RELEASE_PLAN_1.0.md index c830fa4b..fe94a677 100644 --- a/design/RELEASE_PLAN_1.0.md +++ b/design/RELEASE_PLAN_1.0.md @@ -27,7 +27,7 @@ both change the bytes on disk, so both belong in an alpha. | `v1.0-dev` | yes | the first published marker | | `v1.0-alpha` | 2026-08-04 | native format, scan, maintenance, Arrow and Parquet | | `v1.0-alpha2` | 2026-08-18 | object storage, Apache Iceberg, parallel bulk ingest and export | -| `1.0-alpha3` | **not tagged** | retention, `parallel_copy` deduplication, `sort_status` reporting `sorted_kind` | +| `1.0-alpha3` | 2026-09-02 | retention, `parallel_copy` deduplication, `sort_status` reporting `sorted_kind` | The observed cadence is 14 days, from one tag to the next. The dates below keep it. They are a cadence, not a commitment, and each is the date the tag is cut