fix(attachments): pack after manual syncs in the background

[arcaputo3]

Problem

A sync or import sent to the daemon packed attachments inside its own request, with no runtime limit. That pass also listed and re-verified every packed blob in the archive. On an archive with about 29,000 packed blobs (31 GB), each manual sync therefore waited a minute or more after its mail sync to pack a handful of new attachments. On a daemon started on demand at low I/O priority, this took over two minutes on every run.

The obvious alternative, leaving the work to the six-hourly attachment-pack job, does not work for daemons started on demand. They stop after 20 idle minutes, so they may never be running when that job is due.

What changes

• Manual ingests return when their own work is done. After a successful sync, import, or message repair, the daemon marks packing pending and asks the attachment-pack job to run, then returns. The new Scheduler.RequestJob behaves like a cron tick, but is triggered on demand. It returns immediately, a run that is already waiting absorbs the request, and a run in progress gets one follow-up.
• The background pass is bounded and keeps the daemon alive. The job takes the operation gate only after the request releases it. It stops after the job's existing one-minute limit and yields to waiting requests. Like other scheduler work, it counts as daemon activity from the moment it queues, so a daemon started on demand stays up until the pass finishes and then stops as usual. Shutdown cancels it, and committed packs are kept.
• Packing cannot starve. Every successful ingest starts the job, so packing does not depend on the six-hourly tick. When a pass hits its byte budget, it queues a follow-up. When a pass runs out of time after committing progress, it resumes behind waiting work. When a pass runs out of time without committing anything, the work stays pending and the next ingest or tick picks it up. A restarted daemon still checks for blobs left loose on its first pass.
• Each pass verifies a bounded window instead of every packed blob. An attachment-pack pass now verifies one window of at most 128 packed blobs or 32 MiB, using the existing durable cursor. It queues follow-ups only while loose blobs remain. Before, every trigger chained a whole verification cycle (about 1,100 passes on the archive above). The daily attachment-maintenance job still completes the cycle. Part of each pass still grows with the archive: it lists every attachment hash, checks every pack file, and reads the 256 loose-blob shard directories. On the archive above that is about 0.5–2 s per pass.
• Finding new loose blobs no longer reads every attachment row. The candidate queries take hashes from the covering hash indexes and read attachment rows only for unpacked hashes. A query-plan test pins the plan. A seeded differential test checks the results against the row-by-row definition, with mixed-case, NULL, empty, BLOB, and integer hash values.
• Failures reach the user. A background pass has no request to report to. So the next sync, import, or message repair sent to the daemon prints one line on stderr when the job's last pass failed. The line names the error and msgvault pack-attachments. A clean pass clears it.

Behavior notes

• A pack failure no longer appears in the ingest that triggered the pass, because that request has already returned. The daemon logs it, and the next ingest prints the one-line warning above. The ingest result was already independent of packing.
• Scheduled syncs keep their existing behavior. They only mark packing pending.
• Post-removal repack (remove-account, gc, purge-excluded-media) still runs inside the request. It is unchanged here.

Measurements

These were taken on a copy of a real archive: 63 GB SQLite, 154k attachment rows, 28.6k packed blobs (31 GB raw) in 841 packs, plus 7 new loose blobs (12 MB). They ran at foreground I/O priority.

	Before	After
Sync request after its subprocess finished	61.4 s (inline pack)	under 1 ms
Pack pass for the 7 new blobs	61.4 s, verifying every packed blob	3.1 s in the background, one 32 MiB verification window
Unpacked-blob query, cold / warm	1.78 s / 0.25 s	0.10 s / 0.03 s
One attachment-pack trigger	1,117 chained passes, 19 min 37 s	1 pass, 3.2 s

The first row is the time between the sync's own work finishing and the request returning. On a daemon started on demand at background I/O priority, the same inline pass took 2 min 11 s. The daily attachment-maintenance job still chains the full verification cycle.

🤖 Generated with Claude Code

https://claude.ai/code/session_01793XPgu1pMcfXjFd423RNP