Skip to content

feat(background-agent): enforce durable admission#395

Merged
andrei-hasna merged 32 commits into
mainfrom
feat/e-00101-durable-agent-admission
Jul 24, 2026
Merged

feat(background-agent): enforce durable admission#395
andrei-hasna merged 32 commits into
mainfrom
feat/e-00101-durable-agent-admission

Conversation

@andrei-hasna

Copy link
Copy Markdown
Contributor

Why

Background-agent starts could reuse incompatible daemons, silently change requested auth profiles, race idempotent creation against quota checks, and duplicate lifecycle evidence after lost acknowledgements or restart recovery. Those gaps could start the wrong run, exceed capacity, or let stale supervisors mutate a reclaimed generation.

What changed

  • Added one SQLite BEGIN IMMEDIATE create-or-adopt admission transaction with identity validation and live/recoverable capacity accounting.
  • Added fail-closed package, daemon protocol, admission schema, capability, and exact auth-profile checks across CLI, TUI, app server, persisted runs, and recovery.
  • Added replay-safe lifecycle receipts in the existing event journal with unique receipt keys plus bounded redacted diagnostics.
  • Fenced stop, heartbeat, status, orphan, recovery, and cancellation writes by supervisor generation.
  • Added deterministic state, daemon, and app-server regression coverage and documented the contract in codex-rs/background-agent/ARCHITECTURE.md.

Verification

  • git diff --check
  • Redacted staged and commit-range gitleaks scans
  • Focused regression tests are included for create-or-adopt, identity/profile/schema conflicts, capacity, receipt replay/redaction, daemon incompatibility, and stale-generation cancellation.
  • Rust build, format, and test gates run on repository CI/Blacksmith per repository policy.

Task: E-00101
Acceptance scope: codewith-durable-agent-admission-v1

@andrei-hasna
andrei-hasna force-pushed the feat/e-00101-durable-agent-admission branch from 34af9b4 to 5dbd9d1 Compare July 24, 2026 18:37
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Rebase + Blacksmith testbox gate

Rebased feat/e-00101-durable-agent-admission onto origin/main (86b7e78eb, picking up
blacksmith-testbox.yml from #407/#408) and force-pushed with --force-with-lease.
No conflicts; 29 commits replayed; head is now 5dbd9d194. No code changes were made on top of the rebase.

Blacksmith build gate — run 30117708774: conclusion=success
(runner blacksmith-16vcpu-ubuntu-2404, warm_target=false)

just clippy -p codex-state -p codex-background-agent -p codex-app-server -p codex-cli -p codex-tui -- -D warnings \
  && just test -p codex-state -p codex-background-agent
  • clippy -D warnings clean across all five touched crates (codex-state, codex-background-agent,
    codex-app-server, codex-cli, codex-tui), including --tests, in 6m02s.
  • cargo nextest: 384 tests run, 384 passed, 0 skipped — includes the new admission/receipt
    regressions (e.g. partial_admission_recovery_requires_exact_persisted_execution_identity,
    workflow_branch_admission_rejects_stale_generation_and_is_idempotent,
    daemon_start_rejects_incompatible_running_pid_record).
  • gitleaks over origin/main..HEAD: no leaks found.

Review note (follow-up, not blocking this PR)

background_agent_admission_is_ready / claim_background_agent_supervisor_compatible require the
persisted initial-execution snapshot's packageFingerprint to equal
BACKGROUND_AGENT_RUNTIME_COMPATIBILITY_FINGERPRINT, which embeds
env!("CARGO_PKG_VERSION") of codex-background-agent. Both reconcile loops in
background_agent_live.rs just continue when readiness returns false, and
orphan_stale_background_agent_runs only considers rows with supervisor_id IS NOT NULL.

Consequence: a run admitted (status queued, admission-ready) but not yet claimed when the binary
is upgraded becomes permanently unclaimable — and because
count_live_or_recoverable_background_agent_runs_in_tx counts queued + admission-ready rows, each
stranded run permanently consumes one of the DEFAULT_MAX_ACTIVE_BACKGROUND_AGENT_RUNS (8) slots,
so enough upgrade-straddling runs will make every subsequent admission fail with
background_agent_admission_capacity_exceeded.

Suggested follow-up (own PR, keeps this diff scoped): terminalize unclaimed runs whose persisted
package/schema fingerprint no longer matches the running binary (fail with an explicit
admission runtime package incompatible reason + lifecycle receipt) instead of skipping them, so
capacity is released on upgrade.

@andrei-hasna
andrei-hasna force-pushed the feat/e-00101-durable-agent-admission branch 2 times, most recently from e89c980 to 405b5b8 Compare July 24, 2026 20:42
andrei-hasna and others added 5 commits July 25, 2026 00:23
Three review defects in the durable-admission change.

1. Stop persisting recoverable identity plaintext (security regression).
Admission wrote `idempotency_key` and `auth_profile_ref` through a
reversible `codewith-opaque-v1:` hex envelope, chosen so the state
redactor and the local-state secrets doctor would skip those cells. A
credential-shaped value therefore persisted, survived the doctor, and
printed in `agent list --json` — weaker than the merge base, which bound
both through `redact_state_string`. Idempotency keys are now stored only
as a one-way SHA-256 digest; dedupe, replay, and immutable-identity
comparisons run off the digest, preserving byte-exact matching without
keeping plaintext. Auth-profile aliases must survive so the worker can
load the same profile, so they go back through state redaction. Nothing
decodes either column on read. `agent start` also now refuses an
`--auth-profile` alias that config resolution did not select, so an
unvalidated alias cannot reach admission and fail later mid-run.

2. Reclaim admission capacity after a version bump. A claim requires the
persisted admission schema and execution-snapshot `packageFingerprint`
to equal the running binary, but `orphan_stale_background_agent_runs`
only touches rows with a supervisor, while queued/orphaned rows still
count towards `DEFAULT_MAX_ACTIVE_BACKGROUND_AGENT_RUNS`. After an
upgrade those rows were unclaimable forever yet kept burning slots until
every admission failed with capacity exceeded and nothing self-healed.
`terminalize_incompatible_background_agent_runs` now fails such rows
closed with an explicit reason plus a lifecycle receipt, and runs before
every admission and on each supervisor reconcile.

3. Recover from an upgrade with the old daemon still alive.
`ensure_daemon_record_compatible` bailed on any mismatch and gated both
`start()` and `status()`, so replacing the binary while the previous
daemon ran broke `agent start` outright. `start()` now stops and
replaces the mismatched daemon (still never reusing it), and read-only
`status()` reports `incompatibleRunning` with the exact remediation
command instead of erroring.

Adds regression tests for all three and corrects ARCHITECTURE.md, which
claimed state redaction ran before persistence while the code bypassed
it.
prompt_snapshot_ref was bound raw by the durable-admission insert while the
merge base wrote it through redact_state_string, so an admission whose prompt
ref carried credential-shaped material stored it recoverably. Named-column
fixes have now missed a sibling column twice, so route every caller-supplied
string through one projection instead:

- RedactedBackgroundAgentRunColumns produces every string column that
  admission writes to background_agent_runs, and the idempotent-replay
  identity comparison reads the same projection, so write and compare cannot
  drift. prompt_snapshot_ref is redacted (not digested) because the CLI, TUI,
  and app-server surface it on read and replay matches it verbatim.
- RedactedBackgroundAgentExecutionSnapshotColumns does the same for the
  execution-snapshot columns, covering the raw recovery_policy and
  config_fingerprint bindings in both the in-tx and standalone inserts.
- The thread-binding update redacts thread_id, thread_store_kind,
  thread_store_id, and rollout_path for the same reason.

The new regression test asserts the invariant generically rather than per
column: it admits a run whose every caller-supplied string carries the same
secret-shaped token, then scans every cell of every background_agent* table
for that token. Against the previous head it fails with 12 leaking columns
(including prompt_snapshot_ref); it passes here.

ARCHITECTURE.md now states the invariant and classifies every persisted
column as digested, redacted, or provably non-secret.
@andrei-hasna
andrei-hasna force-pushed the feat/e-00101-durable-agent-admission branch from 405b5b8 to ae44923 Compare July 24, 2026 21:26
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Round 3 — plaintext leak closed structurally + full column sweep

Head: ae449238be1d4f3df169699976a59acc1aa26a36 (rebased on origin/main ef40a93)
Blacksmith testbox gate: run 30127720260conclusion=success, headSha=ae449238b, 387/387 tests passed, clippy -D warnings clean across codex-state, codex-background-agent, codex-app-server, codex-cli, codex-tui.

The named-column fix, and why it is no longer named-column

prompt_snapshot_ref was bound raw (the merge base wrote it through redact_state_string). Chosen fix: redact_state_string, not a digest — the ref is surfaced on read (codewith agent show, TUI agent list, AgentStartResponse) and is matched verbatim by idempotent replay, so a one-way digest would break both; redaction restores the merge-base contract exactly and is a no-op for non-secret refs.

Since a named-column fix has now missed a sibling twice, the invariant is enforced in one place instead:

  • RedactedBackgroundAgentRunColumns::from_params produces every string column admission writes to background_agent_runs, and validate_existing_background_agent_admission_in_tx compares stored rows against the same projection — write and compare can no longer drift.
  • RedactedBackgroundAgentExecutionSnapshotColumns::from_params does the same for the execution-snapshot columns (used by the in-tx admission insert, the standalone insert, and the replay comparison).

Column sweep — every write in the background-agent state path

Protection Columns
Digested (one-way) background_agent_runs.idempotency_key; background_agent_runs.admission_identity_sha256; background_agent_lifecycle_receipts.receipt_key / .operation_identity_sha256; background_agent_events.receipt_key; execution-snapshot payload_json.authProfileIdentitySha256
Redacted runs: request_id, source†, prompt_snapshot_ref†, input_snapshot_ref, thread_id†, thread_store_kind†, thread_store_id†, rollout_path†, parent_thread_id†, parent_agent_run_id†, spawn_linkage_json, auth_profile_ref, status_reason, config_fingerprint†, version_fingerprint†. execution_snapshots: snapshot_kind†, payload_json, recovery_policy†, config_fingerprint†. events / lifecycle_receipts / status_snapshots: payload_json (+ summary). Thread-binding UPDATE: thread_id†, thread_store_kind†, thread_store_id†, rollout_path
Provably non-secret id (runtime-generated uuid; PK/FK joining every table), closed enums desired_state / status / retention_state, supervisor process bookkeeping (supervisor_id, generation, pid, pgid, job_id, start_token, stderr_log_path, lease ids — all runtime-generated), integer timestamps/counters/seqs, literal status reasons in the reaper paths

† = raw before this commit; 12 of them were leaking.

Generic regression test

background_agent_admission_persists_no_plaintext_secret_in_any_column admits a run whose every caller-supplied string carries the same sk-proj-… token, then enumerates sqlite_master + pragma_table_info and scans every cell of every background_agent* table for that substring. It is not per-column, so a future raw binding fails it automatically.

  • Against the previous head it FAILS with: execution_snapshots.recovery_policy, execution_snapshots.config_fingerprint, runs.source, runs.prompt_snapshot_ref, runs.thread_id, runs.thread_store_kind, runs.thread_store_id, runs.rollout_path, runs.parent_thread_id, runs.parent_agent_run_id, runs.config_fingerprint, runs.version_fingerprint.
  • Passes here, and additionally asserts the run is still readable, the ref round-trips redacted (not dropped), and idempotent replay still resolves the same run from the raw key.

ARCHITECTURE.md now states the invariant and carries the same classification table.

Defects 2 and 3 (unchanged from round 2, re-verified green on this head)

  • terminalize_incompatible_background_agent_runs fails closed any queued/orphaned run the current binary can never claim, releasing the capacity slot with an explicit reason + lifecycle receipt; wired into the CLI, app-server admission, and the AgentRunStore reconcile. Covered by background_agent_runtime_upgrade_reclaims_stranded_admission_capacity (PASS).
  • Daemon start() stops and replaces an incompatible-but-running daemon (ReplacedIncompatible), and read-only status() reports IncompatibleRunning with the exact remediation command instead of erroring. Covered by daemon_start_replaces_incompatible_running_daemon and daemon_status_reports_incompatible_running_daemon_with_remediation (both PASS).

Pure rustfmt output for the redacted column projection helpers and the
all-cells plaintext test; no behaviour change. Fixes the failing
`cargo fmt --all -- --check` gate on this branch.
@andrei-hasna
andrei-hasna merged commit 54333c9 into main Jul 24, 2026
28 checks passed
@andrei-hasna
andrei-hasna deleted the feat/e-00101-durable-agent-admission branch July 24, 2026 23:10
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 24, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant