prune: add --group-by, apply the retention rules per group - #10291
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #10291 +/- ##
==========================================
+ Coverage 87.52% 87.55% +0.02%
==========================================
Files 103 103
Lines 18628 18676 +48
Branches 2862 2872 +10
==========================================
+ Hits 16305 16352 +47
Misses 1622 1622
- Partials 701 702 +1 ☔ View full report in Codecov by Harness. |
prune applied the retention rules to one flat pool of all matching archives, so unrelated backup sets competed for the same retention slots: with a "home" and an "etc" series in one repository, --keep-daily 7 kept 7 daily archives in total, not 7 per series. The docs worked around this by telling users to run one prune call per series, and in a repository shared by several machines even that was not enough, because different hosts may use the same series name. --group-by KEYS groups the selected archives by the archive attributes name, host, user and/or tags, and applies the full rule set (--from prefilter and keep-oldest included) to each group separately. The default is --group-by name,host, so neither different series nor the same series name used by different hosts compete for retention slots. Archive series names are not unique in a shared repository, so grouping by name alone would still let one machine's archives push another machine's archives out of the retention slots. This can only ever keep more archives than before, never fewer. --group-by "" (or "none") restores the previous behaviour of treating all selected archives as one group, --group-by name groups by the series name only. Note the split between the two mechanisms: NAME / -a select which archives are considered at all, --group-by subdivides those into independent retention pools. Archives without host / user metadata (e.g. transferred from a borg 1.x repo) form their own group. Internal tags (starting with @) do not affect grouping. With more than one group, prune logs a summary line per group, and the JSON output gains a "group" object per archive.
3be9d23 to
7004e87
Compare
|
Force-pushed with two changes: 1. 2. The default is now I had argued for That reasoning was wrong about which failure matters more. Weighing the two:
Data loss beats disk growth, and The epilog now points at Tests updated accordingly: |
Closes the gap discussed in #10288: in a repository shared by several hosts/users, prune could not
express "apply this policy to my archives only".
The problem
pruneapplied the retention rules to one flat pool of all matching archives. With ahomeand anetcseries in one repository,--keep-daily 7kept 7 daily archives in total, not 7 perseries. The docs worked around this by telling users to run one prune call per series — and in a
repository shared by several machines even that was not enough, because different hosts may use the
same series name for their own, unrelated data.
The change
--group-by KEYSgroups the selected archives by the archive attributesname,host,userand/or
tags, and applies the full rule set to each group separately —--fromprefilter andkeep-oldest included.
The two mechanisms stay cleanly separated:
NAME/-adecide which archives are considered at all,--group-bydecides how those are subdivided into independent retention pools.The default is
--group-by name, so archives of different series no longer compete forretention slots.
--group-by ""(or"none") restores the old behaviour.I picked
namerather than something likename,hostfor the default on purpose. The series name ischosen by the user and is therefore stable, while a hostname need not be: with
hostin the default,a container with a random hostname per run would put every archive in a group of its own and
--keep-daily 7would keep everything. Silently never pruning is a worse failure than the onebeing fixed, since nothing errors.
--group-by name,hostis one flag away for those who want it, andthe epilog warns about grouping on an unstable attribute.
Shared-repository usage now reads:
Details
do_prunegroups, then calls the new_compute_keep()once per group.previously_keptisgroup-local by construction.
base_timestampis computed indo_pruneand passed down, so interval-based rules use thesame reference in every group instead of a slightly different
datetime.now()per group.host/usermetadata (e.g. transferred from a borg 1.x repo) get""andform their own group.
@) are excluded from thetagskey, so tagging an archive does notrelocate it into a group of its own.
already say everything, so nothing extra is printed.
"group"object per archive.GroupBySpecreturns the validated string rather than a tuple, likeSortBySpecdoes — theargument parser feeds the parsed value back through the spec, so it has to be idempotent.
Compatibility
This changes default behaviour, but only in the safe direction: grouping can keep more archives
than before, never fewer. Invocations that already select a single series (
NAMEor-a, i.e. whatthe docs recommended) are unaffected.
Suggested CHANGES entry, if you want one — a behaviour change rather than a plain new feature:
I left CHANGES.rst alone since it only ever gets touched in your own "update CHANGES" commits.
Tests — please look at this part
34 existing prune tests now pass
--group-by "". They give each archive its own name(
test-1…test-N) so they can be told apart in the prune output, while testing the rule engine asif all of them belonged to one series — under the new default each landed in a group of one. I
routed them through a
prune_ungrouped()helper whose docstring says exactly that, rather thanweakening any assertion. This is the honest measure of how much the default change shifts, so it is
worth a look during review.
The two visualized examples (
docs/misc/prune-example*.txt) describe a single daily series and staycorrect as written; the quickstart and FAQ examples each select one series and are unaffected.
New tests:
archive_group_key(key order,@-tag exclusion, missing host/user metadata),group_archives,format_group_key,GroupBySpec(invalid key, duplicate key, idempotency)--group-by "",--group-by name,hostin a two-hostshared repo,
--group-by tags, the per-group log lines, the JSONgroupfield, invalid keyFull test suite: 2918 passed, 977 skipped.
ruff checkclean.