borg2: compact wants to remove pack with UNUSED pack_id

[ThomasWaldmann]

I played with #9846 code and run out of disk space while borg create.

Then ran borg check (OK) and borg repo-list (OK), but borg compact crashed:

borg create ...
Failed to release the lock <Store(backend='PosixFS')> (was not locked).

Error:

NotLocked: Failed to release the lock <Store(backend='PosixFS')> (was not locked).

If reporting bugs, please include the following:

Traceback (most recent call last):
  File "/Users/tw/w/borg/src/borg/archiver/create_cmd.py", line 289, in do_create
    create_inner(archive, cache, fso)
  File "/Users/tw/w/borg/src/borg/archiver/create_cmd.py", line 181, in create_inner
    self._rec_walk(
  File "/Users/tw/w/borg/src/borg/archiver/create_cmd.py", line 558, in _rec_walk
    self._rec_walk(
  File "/Users/tw/w/borg/src/borg/archiver/create_cmd.py", line 558, in _rec_walk
    self._rec_walk(
  File "/Users/tw/w/borg/src/borg/archiver/create_cmd.py", line 558, in _rec_walk
    self._rec_walk(
  [Previous line repeated 9 more times]
  File "/Users/tw/w/borg/src/borg/archiver/create_cmd.py", line 492, in _rec_walk
    status = self._process_any(
             ^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archiver/create_cmd.py", line 305, in _process_any
    return fso.process_file(
           ^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archive.py", line 1438, in process_file
    self.process_file_chunks(
  File "/Users/tw/w/borg/src/borg/archive.py", line 1178, in process_file_chunks
    chunk_entry = chunk_processor(chunk)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archive.py", line 1173, in chunk_processor
    chunk_entry = cache.add_chunk(chunk_id, {}, data, stats=stats, ro_type=ROBJ_FILE_STREAM)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/cache.py", line 849, in add_chunk
    self._maybe_write_chunks_cache(now)
  File "/Users/tw/w/borg/src/borg/cache.py", line 869, in _maybe_write_chunks_cache
    write_chunkindex_to_repo(self.repository, self._chunks, clear=clear)
  File "/Users/tw/w/borg/src/borg/cache.py", line 596, in write_chunkindex_to_repo
    stored_hashes = set(list_chunkindex_hashes(repository))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/cache.py", line 521, in list_chunkindex_hashes
    for info in repository.store_list("index"):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/repository.py", line 931, in store_list
    self._lock_refresh()
  File "/Users/tw/w/borg/src/borg/repository.py", line 442, in _lock_refresh
    self.lock.refresh()
  File "/Users/tw/w/borg/src/borg/storelocking.py", line 264, in refresh
    raise LockTimeout(str(self.store))
borg.storelocking.LockTimeout: Failed to create/acquire the lock <Store(backend='PosixFS')> (timeout).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tw/w/borg/src/borg/archiver/_common.py", line 173, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archiver/create_cmd.py", line 234, in do_create
    with Cache(
  File "/Users/tw/w/borg/src/borg/cache.py", line 941, in __exit__
    self.close()
  File "/Users/tw/w/borg/src/borg/cache.py", line 958, in close
    self.security_manager.save(self.manifest, self.key)
  File "/Users/tw/w/borg/src/borg/security.py", line 106, in save
    with SaveFile(self.location_file) as fd:
  File "/Users/tw/w/borg/src/borg/platform/base.py", line 253, in __enter__
    self.tmp_fd, self.tmp_fname = mkstemp_mode(prefix=self.tmp_prefix, suffix=".tmp", dir=self.dir, mode=0o666)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/helpers/fs.py", line 634, in mkstemp_mode
    return _mkstemp_inner(dir, prefix, suffix, flags, output_type, mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/helpers/fs.py", line 590, in _mkstemp_inner
    fd = _os.open(file, flags, mode)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device: '/Users/tw/Library/Application Support/borg/security/d49a1d48ea5ed139e62c7b7ab551cb3ea036dc53f3726eb1ed5d58ffaed5246b/location-2_oijg_n.tmp'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tw/w/borg/src/borg/archiver/__init__.py", line 636, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archiver/__init__.py", line 513, in run
    rc = func(args)
         ^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archiver/_common.py", line 141, in wrapper
    with repository:
  File "/Users/tw/w/borg/src/borg/repository.py", line 404, in __exit__
    self.close()
  File "/Users/tw/w/borg/src/borg/repository.py", line 586, in close
    self.lock.release()
  File "/Users/tw/w/borg/src/borg/storelocking.py", line 219, in release
    raise NotLocked(str(self.store))
borg.storelocking.NotLocked: Failed to release the lock <Store(backend='PosixFS')> (was not locked).

Platform: Darwin MacBook-Pro-2.local 24.6.0 Darwin Kernel Version 24.6.0: Tue Apr 21 20:16:56 PDT 2026; root:xnu-11417.140.69.710.16~1/RELEASE_ARM64_T6030 arm64
Borg: 2.0.0b23.dev221+g15d55d48e.d20260702  Python: CPython 3.11.15 msgpack: 1.2.0 fuse: llfuse 1.5.2 [mfusepy,pyfuse3,llfuse]
PID: 2612  CWD: /Users/tw/w/borg
sys.argv: ['/Users/tw/w/borg-env/bin/borg', 'create', '--list', 'arch', '/System']
SSH_ORIGINAL_COMMAND: None

(borg-env) tw@MacBook-Pro-2 borg % borg check                     
Starting full repository check
Starting from beginning.
Finished checking packs.
Checked 11 index files (0 errors) and 354872 packs (0 errors).
Finished full repository check, no problems found.
Starting archive consistency check...
Analyzing archive arch 2026-07-03 04:52:03.719365+02:00 58be7442f72eee6d3137b19daa3e29218a72eb61df30ade225d050ddb163b131 (1/6)
Analyzing archive arch 2026-07-03 04:52:42.806715+02:00 28bfb7e46d8b1791fd767dbb8daafead4bcd15aa2e7f2a3980b935c1fb7ed04c (2/6)
Analyzing archive arch 2026-07-03 04:53:01.682365+02:00 b4bcf4254a7412852a2a7729a849c97d384edcb55b25398b78f37ec9613fd88b (3/6)
Analyzing archive arch 2026-07-03 04:53:30.236226+02:00 f99329b443184ff05e651c5fb18d555df44360f75d5486024ca5abcd5fa23825 (4/6)
Analyzing archive arch 2026-07-03 04:53:48.438667+02:00 bb64fc8d055db543830d7ee6eea0fd621c3fa876f8bcb65a5e6e7160d5f440c9 (5/6)
Analyzing archive arch 2026-07-03 04:55:22.903040+02:00 bf4b93a7c56c2623a1a32d847b527173300e8ef90d6dac6aed075acfc8a6efbb (6/6)
Archive consistency check complete, no problems found.

(borg-env) tw@MacBook-Pro-2 borg % borg repo-list
58be7442  Fri, 2026-07-03 04:52:03 +0200  arch                         tw          MacBook-Pro-2  
28bfb7e4  Fri, 2026-07-03 04:52:42 +0200  arch                         tw          MacBook-Pro-2  
b4bcf425  Fri, 2026-07-03 04:53:01 +0200  arch                         tw          MacBook-Pro-2  
f99329b4  Fri, 2026-07-03 04:53:30 +0200  arch                         tw          MacBook-Pro-2  
bb64fc8d  Fri, 2026-07-03 04:53:48 +0200  arch                         tw          MacBook-Pro-2  
bf4b93a7  Fri, 2026-07-03 04:55:22 +0200  arch                         tw          MacBook-Pro-2  

(borg-env) tw@MacBook-Pro-2 borg % borg compact
Starting compaction / garbage collection...
Getting object IDs from the cached chunks index...
Computing object IDs used by archives...
Analyzing archive arch 2026-07-03 04:52:03.719365+02:00 58be7442f72eee6d3137b19daa3e29218a72eb61df30ade225d050ddb163b131 (1/6)
Analyzing archive arch 2026-07-03 04:52:42.806715+02:00 28bfb7e46d8b1791fd767dbb8daafead4bcd15aa2e7f2a3980b935c1fb7ed04c (2/6)
Analyzing archive arch 2026-07-03 04:53:01.682365+02:00 b4bcf4254a7412852a2a7729a849c97d384edcb55b25398b78f37ec9613fd88b (3/6)
Analyzing archive arch 2026-07-03 04:53:30.236226+02:00 f99329b443184ff05e651c5fb18d555df44360f75d5486024ca5abcd5fa23825 (4/6)
Analyzing archive arch 2026-07-03 04:53:48.438667+02:00 bb64fc8d055db543830d7ee6eea0fd621c3fa876f8bcb65a5e6e7160d5f440c9 (5/6)
Analyzing archive arch 2026-07-03 04:55:22.903040+02:00 bf4b93a7c56c2623a1a32d847b527173300e8ef90d6dac6aed075acfc8a6efbb (6/6)
Cleaning archives directory from soft-deleted archives...
Deleting 659382 unused objects...    # takes long, risky: index already gone!
Local Exception

Error:

ObjectNotFound: packs/ff/ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

If reporting bugs, please include the following:

Traceback (most recent call last):
  File "/Users/tw/w/borg/src/borg/archiver/__init__.py", line 636, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archiver/__init__.py", line 513, in run
    rc = func(args)
         ^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archiver/_common.py", line 173, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archiver/compact_cmd.py", line 279, in do_compact
    ).garbage_collect()
      ^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg/src/borg/archiver/compact_cmd.py", line 48, in garbage_collect
    self.report_and_delete()
  File "/Users/tw/w/borg/src/borg/archiver/compact_cmd.py", line 168, in report_and_delete
    self.compact_packs()
  File "/Users/tw/w/borg/src/borg/archiver/compact_cmd.py", line 261, in compact_packs
    self.repository.store_delete("packs/" + bin_to_hex(pid))
  File "/Users/tw/w/borg/src/borg/repository.py", line 947, in store_delete
    return self.store.delete(name, deleted=deleted)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg-env/lib/python3.11/site-packages/borgstore/store.py", line 478, in delete
    self._backend_call(lambda: self.backend.delete(nested_name), key="delete", volume=0)
  File "/Users/tw/w/borg-env/lib/python3.11/site-packages/borgstore/store.py", line 283, in _backend_call
    result = operation()
             ^^^^^^^^^^^
  File "/Users/tw/w/borg-env/lib/python3.11/site-packages/borgstore/store.py", line 478, in <lambda>
    self._backend_call(lambda: self.backend.delete(nested_name), key="delete", volume=0)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tw/w/borg-env/lib/python3.11/site-packages/borgstore/backends/posixfs.py", line 276, in delete
    raise ObjectNotFound(name) from None
borgstore.backends.errors.ObjectNotFound: packs/ff/ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

Platform: Darwin MacBook-Pro-2.local 24.6.0 Darwin Kernel Version 24.6.0: Tue Apr 21 20:16:56 PDT 2026; root:xnu-11417.140.69.710.16~1/RELEASE_ARM64_T6030 arm64
Borg: 2.0.0b23.dev221+g15d55d48e.d20260702  Python: CPython 3.11.15 msgpack: 1.2.0 fuse: llfuse 1.5.2 [mfusepy,pyfuse3,llfuse]
PID: 11223  CWD: /Users/tw/w/borg
sys.argv: ['/Users/tw/w/borg-env/bin/borg', 'compact']
SSH_ORIGINAL_COMMAND: None

[ThomasWaldmann]

Note: borg repo-list after that rebuilt the index by scanning packs.

But this is a practical example of how risky it is to first kill the index, then do lots of stuff (deleting 600.000 packs) and crash before writing the new index.

[ThomasWaldmann]

Fixed — verified this no longer reproduces.

The crash was compact calling store_delete("packs/ff…ff") for a pack whose id is the UNKNOWN_BYTES32 filler (an unresolved / still-pending chunk-index entry left behind when borg create ran out of disk space). Two independent fixes now cover it:

1. Root cause — create: do not wrap repository writes in backup_io("read") (silent data loss on ENOSPC) #9853: borg create no longer swallows a repository ENOSPC as a per-file read: warning and commits anyway. It was wrapping add_chunk()'s repository writes in backup_io("read"); that's removed, so a repository IO failure is now critical and aborts create before archive.save(). The out-of-space create that produced the stale/unresolved index entries in the first place no longer leaves that state behind.

2. Symptom / belt-and-suspenders — borg2: crash while compact #9850 (commit 1930c0c): compact_packs() now wraps store_delete() in try/except StoreObjectNotFound, so a stale chunk-index entry referencing a non-existent pack (including the ff…ff filler id) is tolerated with a Pack … to delete was already gone. warning instead of crashing.

Verified: injecting a chunk-index entry with pack_id = ff…ff and running borg compact now logs the "already gone" warning, completes cleanly, and borg check reports no problems. Under a create/delete/compact churn on a space-limited filesystem, out-of-space creates now fail cleanly (no committed archive, no unresolved ff…ff entries persisted).

Closing as fixed by #9850 and #9853.