Skip to content

prune: add --group-by, apply the retention rules per group - #10291

Merged
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:prune-group-by
Aug 31, 2026
Merged

prune: add --group-by, apply the retention rules per group#10291
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:prune-group-by

Conversation

@ThomasWaldmann

Copy link
Copy Markdown
Member

Closes the gap discussed in #10288: in a repository shared by several hosts/users, prune could not
express "apply this policy to my archives only".

The problem

prune applied the retention rules to one flat pool of all matching archives. With a home and an
etc series in one repository, --keep-daily 7 kept 7 daily archives in total, not 7 per
series. The docs worked around this by telling users to run one prune call per series — and in a
repository shared by several machines even that was not enough, because different hosts may use the
same series name for their own, unrelated data.

The change

--group-by KEYS groups the selected archives by the archive attributes name, host, user
and/or tags, and applies the full rule set to each group separately — --from prefilter and
keep-oldest included.

The two mechanisms stay cleanly separated:

  • NAME / -a decide which archives are considered at all,
  • --group-by decides how those are subdivided into independent retention pools.

The default is --group-by name, so archives of different series no longer compete for
retention slots. --group-by "" (or "none") restores the old behaviour.

I picked name rather than something like name,host for the default on purpose. The series name is
chosen by the user and is therefore stable, while a hostname need not be: with host in the default,
a container with a random hostname per run would put every archive in a group of its own and
--keep-daily 7 would keep everything. Silently never pruning is a worse failure than the one
being fixed, since nothing errors. --group-by name,host is one flag away for those who want it, and
the epilog warns about grouping on an unstable attribute.

Shared-repository usage now reads:

borg prune --group-by name,host --keep-daily 7 --keep-weekly 4   # every series of every host, in one call
borg prune -a host:myhost --keep-daily 7 --keep-weekly 4         # only my archives, default grouping per series

Details

  • do_prune groups, then calls the new _compute_keep() once per group. previously_kept is
    group-local by construction.
  • One base_timestamp is computed in do_prune and passed down, so interval-based rules use the
    same reference in every group instead of a slightly different datetime.now() per group.
  • Archives without host / user metadata (e.g. transferred from a borg 1.x repo) get "" and
    form their own group.
  • Internal tags (starting with @) are excluded from the tags key, so tagging an archive does not
    relocate it into a group of its own.
  • With more than one group, prune logs a summary line per group; with a single group the totals
    already say everything, so nothing extra is printed.
  • The JSON output gains a "group" object per archive.
  • GroupBySpec returns the validated string rather than a tuple, like SortBySpec does — the
    argument parser feeds the parsed value back through the spec, so it has to be idempotent.

Compatibility

This changes default behaviour, but only in the safe direction: grouping can keep more archives
than before, never fewer. Invocations that already select a single series (NAME or -a, i.e. what
the docs recommended) are unaffected.

Suggested CHANGES entry, if you want one — a behaviour change rather than a plain new feature:

  • prune: added --group-by, the retention rules are now applied per group of archives, grouped
    by archive series name by default. Previously all matching archives competed for the same
    retention slots. Use --group-by "" for the old behaviour, or e.g. --group-by name,host
    in a repository shared by multiple machines.

I left CHANGES.rst alone since it only ever gets touched in your own "update CHANGES" commits.

Tests — please look at this part

34 existing prune tests now pass --group-by "". They give each archive its own name
(test-1test-N) so they can be told apart in the prune output, while testing the rule engine as
if all of them belonged to one series — under the new default each landed in a group of one. I
routed them through a prune_ungrouped() helper whose docstring says exactly that, rather than
weakening any assertion. This is the honest measure of how much the default change shifts, so it is
worth a look during review.

The two visualized examples (docs/misc/prune-example*.txt) describe a single daily series and stay
correct as written; the quickstart and FAQ examples each select one series and are unaffected.

New tests:

  • unit: archive_group_key (key order, @-tag exclusion, missing host/user metadata),
    group_archives, format_group_key, GroupBySpec (invalid key, duplicate key, idempotency)
  • integration: default grouping by name, --group-by "", --group-by name,host in a two-host
    shared repo, --group-by tags, the per-group log lines, the JSON group field, invalid key

Full test suite: 2918 passed, 977 skipped. ruff check clean.

@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.38710% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.55%. Comparing base (d4a11c9) to head (7004e87).
⚠️ Report is 3 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/borg/archiver/prune_cmd.py 97.95% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #10291      +/-   ##
==========================================
+ Coverage   87.52%   87.55%   +0.02%     
==========================================
  Files         103      103              
  Lines       18628    18676      +48     
  Branches     2862     2872      +10     
==========================================
+ Hits        16305    16352      +47     
  Misses       1622     1622              
- Partials      701      702       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Comment thread src/borg/archiver/prune_cmd.py Outdated
prune applied the retention rules to one flat pool of all matching archives, so
unrelated backup sets competed for the same retention slots: with a "home" and
an "etc" series in one repository, --keep-daily 7 kept 7 daily archives in
total, not 7 per series. The docs worked around this by telling users to run
one prune call per series, and in a repository shared by several machines even
that was not enough, because different hosts may use the same series name.

--group-by KEYS groups the selected archives by the archive attributes name,
host, user and/or tags, and applies the full rule set (--from prefilter and
keep-oldest included) to each group separately.

The default is --group-by name,host, so neither different series nor the same
series name used by different hosts compete for retention slots. Archive series
names are not unique in a shared repository, so grouping by name alone would
still let one machine's archives push another machine's archives out of the
retention slots. This can only ever keep more archives than before, never fewer.
--group-by "" (or "none") restores the previous behaviour of treating all
selected archives as one group, --group-by name groups by the series name only.

Note the split between the two mechanisms: NAME / -a select which archives are
considered at all, --group-by subdivides those into independent retention
pools.

Archives without host / user metadata (e.g. transferred from a borg 1.x repo)
form their own group. Internal tags (starting with @) do not affect grouping.

With more than one group, prune logs a summary line per group, and the JSON
output gains a "group" object per archive.
@ThomasWaldmann

Copy link
Copy Markdown
Member Author

Force-pushed with two changes:

1. defaultdict in group_archives(), per the review comment.

2. The default is now --group-by name,host (was name).

I had argued for name on the grounds that the series name is user-chosen and therefore stable,
while a hostname need not be — with host in the default, a container with a random hostname each
run puts every archive in a group of its own and --keep-daily 7 keeps everything.

That reasoning was wrong about which failure matters more. Weighing the two:

  • name default, shared repository: prune deletes another host's archives, because their home
    series competes with yours for the same retention slots. Data loss, and it is the exact problem
    this PR set out to fix — a default of name only fixes it for an explicitly filtered prune.
  • name,host default, unstable hostname: prune keeps too much. Disk usage, plainly visible in
    repo size and in the per-group summary lines, and one flag (--group-by name) away from a fix.

Data loss beats disk growth, and name,host also keeps the same "only ever keeps more, never fewer"
safety property relative to today's behaviour. It additionally makes prune agree with
#10292, where create --group-by defaults to name,host for picking the archive whose
files cache is reused — having the two commands disagree about what an archive's group is would be
hard to explain.

The epilog now points at BORG_HOSTNAME and --group-by name for the unstable-hostname case, and
mentions that --info prints one summary line per group, which makes that situation visible.

Tests updated accordingly: test_prune_groups_by_name_and_host_by_default and
test_prune_group_by_name_only now use a shared-repo fixture (host1 with home + etc, host2 with
its own home) so both the series and the host dimension are covered. Full suite: 2918 passed, 977
skipped.

@ThomasWaldmann
ThomasWaldmann merged commit 35872ae into borgbackup:master Aug 31, 2026
24 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the prune-group-by branch August 31, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant