feat: enforce scheduled content retention by partition drop

[pjb157]

Summary

Implements the data retention policy for every class of customer content the API stores. Scheduled deletion is partition drop only — content lives in time-keyed partitions and expires by irreversibly dropping whole relations through one shared, journaled, crash-safe engine. There is no recurring row-by-row content deletion anywhere after this PR, and every destructive control ships disabled.

How each data class is deleted

Data class	Storage	Deletion mechanism
Synchronous/batchless response content (requests, dedicated input templates, response steps)	Daily retained_response_objects partitions, keyed by deletion date	Whole response graphs move atomically out of the live tables into the day they become deletable; the day is dropped once PostgreSQL's own UTC clock passes it
Batch response content	Existing weekly batch_requests_archive partitions	A week drops once every batch in it is fully archived, frozen, and individually past its finalization-anchored retention period; completion stamps batches.retention_expired_at in the same transaction
Batch input content (uploaded request payloads)	New weekly request_templates_g2 partitions; new writes cut over behind a flag, the legacy heap freezes in place untouched	A week drops once it is past the horizon and no live file still owns templates in it; the frozen legacy heap is dropped later as one relation in a separately approved forward migration
Batch / file metadata (names, statuses, timestamps, billing)	Ordinary metadata tables	Never deleted by schedule. Files are tombstoned (retention_expired_at, row survives) once every referencing batch's content is gone; batches are stamped at partition drop. Account-lifetime retention, exactly as the policy states
On-request erasure	All of the above	Unchanged and immediate: whole-graph deletion, file/creator erasure, and the orphan purge — which now requires an explicit-deletion tombstone on every branch so scheduled expiry can never leak through it

One retirement engine, three families

Every partition drop runs through a single state machine (partition_retirement.rs): journal the exact partition identity (schema + OID, parent OID, child name + OID, bounds — with a CHECK constraint that makes tampered identities unrepresentable), fence the bucket retiring in the same commit so reads fail closed, DETACH PARTITION CONCURRENTLY, FINALIZE recovery after crashes, drop only the journaled relation, and complete atomically. Families differ only in a declarative spec: names, bounds width, eligibility SQL, and an optional metadata stamp. Lock/statement timeouts are retryable no-ops on a durable journal; renamed, re-bounded, or replaced relations are refused; an unfinished journal remains recoverable even after its flag or period is withdrawn.

Destructive DDL runs only on an explicitly installed single-session maintenance pool (max 1 connection), attested at startup against the primary, with server-side lock and statement timeouts.

Reads and writers

Point, list, and count reads are byte-identical before and after content moves; everything in a retiring/retired bucket answers not-found before any physical DDL. Template reads are generation-transparent through one view (the claim path resolves generation-2 ids through a route oracle that prunes to a single weekly partition). Late writers to moved or dropped response identities are blocked by durable content-free fences; claim, pending, and mutation paths remain live-only.

Safety defaults and flags (all off)

batchless_archive_sweep_enabled, batchless_archive_backfill_enabled, retained_response_retirement_enabled, batch_archive_retirement_enabled + batch_archive_retention_days, template_generation_writes_enabled, template_retirement_enabled + template_retention_days. Retention periods have no defaults — enabling any retirement without an explicit positive period fails startup validation, as does enabling retirement without the maintenance endpoint on a dedicated database, or template retirement without the write cutover.

Observability and evidence

Aggregate-only metrics with fixed labels for every phase (movement counts/bytes, partition runway and readiness, per-family retirement and retry counters, route/fence cleanup counters, file-content expiry). The retirement journal and bucket tombstones are permanent, dated, content-free records of every deletion — audit evidence by construction. No request identifiers, owners, models, or payloads appear in any log, metric, or error (enforced by the repo's no-payload-logging guard). A read-only preflight script verifies index readiness and exact partition attachment before any enablement.

Testing

• Full workspace suite green; dedicated integration suites per family (daily response retirement, weekly batch archive, weekly templates) covering crash points, identity fail-closed refusal, recovery without selection flags, reference-gate blocking (live/split/unfrozen batches, live files, unowned rows), metadata stamping idempotence, and bounded cleanup.
• Fresh migration up/down/up cycles across all three retention migrations; every down migration fails closed while lifecycle state exists; live-table relations are never scanned, rewritten, or locked by any migration.
• Checksums cover up and down migrations; sqlx offline metadata verified; lint/clippy/fmt clean.

Rollout (each step independently reversible until noted)

1. Merge; deploy. Nothing changes at runtime — all flags off, expand-only schema.
2. Build the candidate index concurrently (standalone operation); verify with the migration-owned readiness guard and the preflight script.
3. Enable batchless movement with minimal budgets → backfill → legacy drain (rollback: flags off; readers stay archive-aware).
4. Enable daily response retirement (first drop is the point of no return for that day's content — by design).
5. Enable weekly batch-archive retirement with the policy period.
6. Enable the template write cutover (rollback: flag off; new writes return to the legacy heap).
7. Enable template retirement with the policy period; file tombstoning begins releasing weeks.
8. After the legacy heap's full horizon passes and the batchless drain is complete: survivor copy + one approved forward migration drops the legacy template relation.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying control-layer with    Cloudflare Pages

Latest commit:	03a0d3f
Status:	✅  Deploy successful!
Preview URL:	https://b4faad77.control-layer.pages.dev
Branch Preview URL:	https://peter-retention-lifecycle.control-layer.pages.dev

View logs

[Copilot]

Pull request overview

Adds policy-driven, disabled-by-default content retention to the fusillade daemon and Postgres storage backend, exposing a configurable sweep worker that expires files, ages out terminal batches, and deletes/redacts eligible batchless requests while emitting aggregate metrics.

Changes:

• Introduces retention policy/cutoff/outcome types in fusillade-core and re-exports them through fusillade and fusillade-arsenal.
• Adds a retention sweep worker to the daemon with startup validation and bounded per-tick chunking.
• Implements Postgres retention sweep logic plus a migration for retention-related access-path indexes and corresponding operator documentation/config validation.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
fusillade/src/manager/mod.rs	Re-exports retention sweep types from fusillade-core.
fusillade/src/lib.rs	Publicly re-exports retention sweep types from the manager module.
fusillade/src/daemon/transitions.rs	Updates test storage stub to satisfy new DaemonStorage retention API.
fusillade/src/daemon/mod.rs	Adds retention startup validation, shutdown-aware helper, and retention sweep background task with metrics.
fusillade/src/daemon/config.rs	Adds retention policy + sweep interval fields to daemon config and round-trip tests.
fusillade/README.md	Documents automated content retention behavior and operational rollout steps.
fusillade-core/src/manager.rs	Defines RetentionSweepPolicy, immutable cutoffs, sweep outcome, and extends DaemonStorage.
fusillade-core/src/lib.rs	Re-exports retention sweep types from manager.
fusillade-core/src/daemon_record/transitions.rs	Updates test storage stub to satisfy new retention API.
fusillade-arsenal/src/postgres.rs	Implements Postgres retention sweeping plus lock-ordering changes and extensive regression tests.
fusillade-arsenal/src/lib.rs	Re-exports retention sweep types from fusillade-core.
fusillade-arsenal/migrations/20260813000000_add_retention_sweep_indexes.up.sql	Adds (non-concurrent) creation of candidate retention indexes with precreate guidance in comments.
fusillade-arsenal/migrations/20260813000000_add_retention_sweep_indexes.down.sql	Drops the new retention sweep indexes.
dwctl/src/config.rs	Wires retention config into dwctl, adds validation, and tests for config invariants.
.github/fixtures/fusillade-migration-sha384.txt	Updates migration checksum fixture to include the new migration.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[pjb157]

Re-review follow-up is available in 38678f83.

• Canceled in-flight redaction now resolves the directly linked model-call step to its canonical response-chain head and scrubs the head plus every descendant, including tool-call rows whose request_id is intentionally NULL.
• Candidate discovery uses the same chain traversal, so content written into a descendant after an earlier redaction is found and scrubbed on the next sweep.
• The real-Postgres regression was observed failing with descendant argument/result JSON intact before the fix, and now passes for both the initial and repeated sweep.
• Refreshed the SHA-384 fixture for the new, unreleased retention-index migration after its rollout documentation changed.

Verification:

• cargo test -p fusillade-arsenal retention_sweep — 8 passed
• cargo test -p fusillade-core retention — 3 passed
• cargo test -p fusillade retention — 2 passed
• just lint rust -- -D warnings — passed against a fresh Postgres schema, including formatting, workspace clippy, payload-logging guard, SQLx prepare check, migration checksums, and repository contract scripts
• git diff --check — passed