Skip to content

OPERATIONS.md

Latest commit

 

History

History
300 lines (253 loc) · 21.6 KB

OPERATIONS.md

File metadata and controls

300 lines (253 loc) · 21.6 KB
Raw

AgentVisor AI Operations Guide

This document is the operator's tour of the harness. It covers boot, health probes, upgrade, key rotation, spool recovery, and the metrics you should alert on.

Boot

agentvisord reads its TOML from $AV_CONFIG (if set), then ./agentvisor.toml, ./config/harness.toml, and $HOME/.agentvisor/agentvisor.toml. The example file (config/harness.example.toml) is NOT searched.

On boot the harness logs a dedicated event on the trust_anchor target:

signer_key_id=<32-hex fingerprint>
signer_public_key_hex=<64-hex Ed25519 public key>
signer_seed_path=<path the seed was loaded from>
freshly_generated=<true only when the seed was created this boot>

Every restart re-emits this pair unconditionally: the trust_anchor target is pinned to INFO inside the process regardless of RUST_LOG, so it survives RUST_LOG=warn, error, and even off. When the seed is freshly generated at startup (no seed file was present at the configured path), a WARN is added on the same always-on target: this is normal for a first-run container but is a hard escalation for a persistent deployment — you've lost your seed file and the new key won't verify old receipts.

Pipe these lines into your log store and diff them across restarts; any change is a rotation event that must be reflected in the downstream trust store.

Health probes

Three routes with distinct semantics:

Route Purpose Failure mode
/health Legacy constant 200 (kept for backwards compatibility with pre-/livez/readyz deployments; identical to /livez today). Only fails when the axum runtime is dead; prefer /livez//readyz for new probes so the semantic split is explicit.
/livez Kubernetes-style liveness. Returns 200 constant. Only fails when the axum runtime is dead; use as livenessProbe.
/readyz Kubernetes-style readiness. 200 while accepting new traffic; 503 during drain. Fails immediately when SIGTERM is received AND while the spool is not writable (probe file create+remove — a full disk or read-only remount flips readiness even though the directory stays readable). Use as readinessProbe.

readinessProbe is the one that turns off the tap during pod deregistration — flip a request-counter graph and you'll see the last 5 seconds of traffic drop off before the process actually terminates.

The included Kubernetes manifest at deploy/kubernetes/agentvisor-ai.yaml wires liveness and readiness (via startupProbe, readinessProbe, and livenessProbe against /livez and /readyz) and uses the daemon's own readiness-controlled pre-drain window (ConfigMap key shutdown_ready_drain_s = 5) instead of a preStop shell hook — the shipped runtime base image (chainguard/glibc-dynamic) is distroless and has no /bin/sh / /bin/sleep, so a preStop: exec: ["sh", "-c", "sleep 5"] hook would have failed FailedPreStopHook and kubelet would have proceeded straight to SIGTERM. /health exists only for pre-/livez//readyz deployments and mirrors /livez — new deployments should probe /livez and /readyz directly.

Shutdown

agentvisord shuts down cleanly on SIGTERM. The order is:

  1. Set the draining flag → /readyz starts returning 503.
  2. If shutdown_ready_drain_s > 0 (default 0), keep ACCEPTING connections for that long so an external load balancer polling /readyz actually observes the 503 before the listener closes. Without it, step 3 begins immediately and a fresh readiness probe sees connection-refused instead. This applies to every deployment target — Kubernetes distroless images (like the shipped chainguard/glibc-dynamic runtime) cannot use a preStop shell hook, and docker-compose / systemd / bare-LB deployments have no preStop equivalent at all. Set this to your LB's poll interval plus one reconciliation (5 s covers a typical kube-proxy / ingress reconciliation and matches the shipped ConfigMap).
  3. Await axum's graceful shutdown (in-flight requests complete; new TCP accepts are refused).
  4. Abort background tasks (reconciler tick, retention sweep, etc.).
  5. Flush and close the state store, broker, and Bridge.

Step-3 tail latency that exhausts the drain budget shows up as av_http_shutdown_drain_timeouts_total. Alert on any increase — it usually means a request was stuck waiting on the upstream and the worker pool couldn't drain before the timeout. The pre-drain window counts against the orchestrator's kill grace period ON TOP of the drain budget.

Key rotation

  1. Generate a new seed: avctl keygen --output signer.new.seed.
  2. Publish its public key to your downstream trust store BEFORE the next step. Use avctl pubkey --seed signer.new.seed to derive public_key_hex and key_id without exposing the seed.
  3. Replace the running seed file atomically (mv, not cp).
  4. SIGHUP is not wired for reloading; you must restart the process. Rolling restarts across a fleet are fine — each pod emits a startup log with the new key id.
  5. The av_receipts::keys::Keyring on the verifier side must have the new key added; old receipts continue to verify against the old key until they age past your retention.

Spool recovery

The reconciler runs every reconcile_tick_s seconds and handles:

  • Pending-close sessions: an ATIF spool file exists but the close bridge event wasn't emitted (crash mid-close). Recovery re-adopts the on-disk journal, emits SESSION_CLOSE, and finalizes.
  • Orphan .json files (round 51 §8.5): a .json with no .atif-auth sidecar is quarantine-renamed to <name>.json.corrupt-<uid> — but only after (a) age > 60s and (b) the stem is not that of a currently-open session. The live-stem check closes the race where a mid-close write is quarantined out from under the finalize path.
  • Sealed pair retention (round 51 §8.1): when atif_retention_days is set, sealed pairs are pruned every hour. Unpaired remnants are LEFT for the orphan sweep. The same sweep is available manually as avctl spool-prune --spool <atif_spool_dir> --retention-days N (external cron, one-off reclaims).
  • Signed-workflow recovery: if a signed session's receipt is missing but its journal is intact, recovery re-signs and emits.

Recovery is idempotent — a killed-then-restarted-then-killed pod does the same work each tick without duplicating events.

Console sync

avctl console-sync uploads durable local spool evidence to the hosted console ingest API. It reads ATIF trajectories from atif_spool_dir (default spool/atif), skips archived collision artifacts, posts one session upsert plus ordered event batches of at most 500 events, then uploads signed receipts from receipts/. When --bridge-dir is set it also tails the embedded Bridge topic logs after the ATIF pass so signed workflow sessions (which have receipts but no ATIF trajectory) publish their durable OCSF events.

avctl console-sync --spool-dir /var/lib/agentvisor/spool/atif \
  --bridge-dir /var/lib/agentvisor/data/bridge \
  --console-url https://console.example.com \
  --deployment dep_123 \
  --token-file /run/secrets/av-console-token

Configuration precedence is CLI flags, then AV_CONSOLE_URL, AV_CONSOLE_DEPLOYMENT, AV_CONSOLE_TOKEN, AV_CONSOLE_BRIDGE_DIR, then an optional ~/.agentvisor/console.toml file (or legacy [console] table in the effective agentvisor.toml):

[console]
url = "https://console.example.com"
deployment = "dep_123"
token_file = "/run/secrets/av-console-token"
bridge_dir = "/var/lib/agentvisor/data/bridge"

The token is never printed; token files are read with trailing whitespace trimmed. Replays are idempotent: the state file (--state-file, default .console-sync-state.json inside the spool) records each session's last synced event sequence and whether its receipt was accepted, plus the last synced Bridge offset per topic/partition. The state file is bound to the console URL and deployment that produced it; use a separate --state-file when syncing to a different destination. ATIF-synced session ids are recorded so Bridge events for unsigned sessions are not duplicated. Receipts are posted only after all events visible in the current pass for that session (ATIF or Bridge) are acknowledged, because sealing rejects later event uploads. Use --dry-run to count what would be sent, or --watch --interval 30 to poll until Ctrl-C.

Bridge OCSF topics map to ingest events as follows:

Bridge topic Ingest kind Notes
agent.session sys opened sets openedAt; closed carries workflow.
agent.tool_call tool / block Allowed calls increment addToolsAllowed; blocked/denied calls increment addToolsBlocked and blocked payout micros when present.
agent.stop_reason guard Stop reason tag/body preserve the OCSF reason.
agent.compression audit Body includes metrics including pruned token count when present.
agent.identity audit Identity validation evidence.
agent.receipt skipped Receipts are uploaded separately from the receipt spool.

avctl setup can create the console config for you. At the end of the guided setup, answer yes to "Connect to the hosted console?", accept or enter the console URL, paste the deployment id and ingest token, then run:

avctl console-sync --watch

Releasing a quarantined session id

A SIGKILL / OOMKill / node loss mid-request leaves a sealed marker under <atif_spool_dir>/inflight-responses/. On restart the reconciler quarantines that session ("quarantining sessions with incomplete effects") and the id returns 400 session is already closed from then on — by design, because the provider may have observed a request whose response was never captured, and silently resuming would put an unattested turn in the audit chain.

If an agent derives its session id from a stable conversation id, that conversation is locked out until an operator intervenes. The manual release procedure:

  1. Confirm the turn's outcome out of band (provider dashboard, upstream logs). You are asserting the uncaptured response either never happened or is acceptable to lose from the trail.
  2. Stop routing traffic for that session id (or drain the pod).
  3. Delete the session's marker file under <atif_spool_dir>/inflight-responses/ — filenames are sha256(session_id:attempt_id)[..32].json, so a stuck session may have MULTIPLE markers (one per response attempt). Grep the directory for a session_id field match inside each file (they are small JSON blobs) rather than computing the digest by hand. Keep a copy if your compliance posture requires evidence of the intervention.
  4. Restart the harness (or wait one reconcile tick). The quarantine set is rebuilt from the markers, so the id admits traffic again.

Do NOT delete markers as routine hygiene — each one is the only evidence that a crash window may contain an unaudited provider interaction.

Metrics you should alert on

Metric Alert condition What it means
av_atif_retention_pruned_total Sudden increase Retention sweep ran (usually just noise, but confirm the atif_retention_days you set).
av_reconciler_last_tick_completed_seconds time() - value > 5 × tick interval The reconciler tick is stalled (hung filesystem, lifecycle-lock deadlock) or has never completed since boot. Closes, promotion retries, idle finalization and outbox replay are all stopped with it. The series exists from boot and reads 0 until the first tick completes.
av_atif_recovery_skipped_total{reason="unauthenticated"} Rate > 0 sustained Someone/something is planting sidecar-less .json files. Escalate.
av_atif_recovery_skipped_total{reason="too_large"} Rate > 0 Adversarial ATIF payload attempts (> MAX_ATIF_RECOVERY_BYTES). Investigate the spool contents.
av_atif_recovery_skipped_total{reason="read_error"} Rate > 0 sustained Recovery failed to open a spool file after MAX_ATIF_RECOVERY_BYTES metadata pre-check — disk / permission fault, or a symlink-plant beat the pre-check. Investigate the disk.
av_atif_recovery_skipped_total{reason="invalid_json"} Any occurrence ATIF file bytes could not parse as JSON. Adversarial payload or bit-rot. Escalate.
av_atif_recovery_skipped_total{reason="nonconformant"} Any occurrence ATIF parsed as JSON but failed strict-mode structural validation. Same class as invalid_json — either a hostile plant or a schema-version mismatch that the operator needs to reconcile. Escalate.
av_atif_recovery_skipped_total{reason="provenance"} Any occurrence .atif-auth sidecar HMAC failed to verify against the current signer key. This is a cryptographic integrity failure — either a key rotation that missed the retirement window, or a tampered/planted trajectory. Escalate immediately.
av_incomplete_sessions_total Sudden increase Sessions where capture_failed was set on the audit chain. After round 51 §6.2 (D7) this is genuinely rare and represents unrecoverable pipeline state, not client disconnects. Escalate.
av_events_dropped_total{stage="response_slot"} Rate > 0 sustained Response capture backpressure — the worker pool is saturated or the state store is slow. Scale up workers or investigate the state backend.
av_http_shutdown_drain_timeouts_total Any increase A graceful drain hit its timeout with requests still in flight; usually upstream latency.
av_ephemeral_close_failures_total Any increase The auto-close spawned for a stock-SDK one-shot session (no X-AV-Session header — the OpenAI-compatibility contract auto-closes it after the response) failed. The session is stranded open until the idle sweeper (session_idle_close_s) reaps it, delaying the receipt. If this fires steadily, the finalizer is unhealthy — investigate.
av_stream_abort_close_failures_total Any increase A background close spawned from a stream-abort drop path failed. Same class as ephemeral-close: the session is left open until the idle sweeper. Steady-state 0 on healthy nodes.
av_stream_abort_no_runtime_total Any increase A stream-abort drop path ran with no live Tokio runtime (harness shutdown, blocking-thread drop). Capture is marked failed so the reconciler retries on next tick. Any increase after boot suggests unclean shutdown ordering.
av_reconciler_panics_total Any increase The reconciler tick body panicked and was caught. The reconciler continues on the next tick, but a panic here indicates a bug or an environmental fault (allocator OOM, disk corruption); investigate the coincident logs.
av_worker_shard_panics_total Any increase A worker shard job body panicked and was caught. The shard continues serving subsequent jobs, but any occurrence is a bug or a hostile input that reached beyond the parser guards; capture the coincident session for triage.
av_bridge_maintenance_panics_total Any increase The bridge maintenance loop tick body panicked. The loop continues on the next interval. Same triage class as reconciler panics.
av_bridge_maintenance_errors_total Rate > 0 sustained The bridge maintenance loop tick returned an error (retention rewrite failed, cold-outbox stage failed). Steady errors indicate a persistent broker/disk fault.
av_bridge_maintenance_join_errors_total Any increase The bridge maintenance loop's spawned task failed to join (panic or cancellation). Any increase after boot is a signal of shutdown-ordering issues or a hostile panic in the maintenance body.
av_atif_retention_panics_total Any increase The hourly ATIF retention sweep tick body panicked and was caught. The loop continues on the next hour, but retention is the ONLY in-process reclaim path — a repeated firing lets the spool grow unbounded until the disk fills. Same triage class as av_bridge_maintenance_panics_total.
av_atif_retention_errors_total Rate > 0 sustained The hourly ATIF retention sweep returned an error (permission drift on the spool subtree, a corrupt sidecar the pass keeps re-tripping on). Steady errors here mean the spool is not being reclaimed; investigate before the disk fills.
av_tcp_nodelay_failures_total Sustained rate > 0 TCP_NODELAY setsockopt failed on an accepted connection. Single events are noise (ECONNRESET race between accept and setsockopt); a sustained rate is either (a) a client half-open flood / SYN flood exhausting sockets before setsockopt runs, or (b) an embedded target without full socket-option support. Under (a) the log is rate-limited to one warn! per process to avoid a log storm — this counter is the primary observability signal instead.
av_stream_abort_panics_total Any increase A background close spawned from the stream-abort drop path PANICKED (caught). The session is left open until the idle sweeper. Same class as av_stream_abort_close_failures_total but for panics rather than returned errors.
av_ephemeral_close_panics_total Any increase Same as av_ephemeral_close_failures_total but for panics rather than returned errors — the auto-close of a completed one-shot ephemeral session panicked in a Drop-spawned task.
av_admission_refund_panics_total Any increase The admission-debit refund closure panicked inside its Drop-spawned spawn_blocking refund. The session and principal ledgers RETAIN the debit; a repeated firing drains the principal budget under real load. Investigate the state backend (Redis) and the store trait implementors' allocator health.
av_idle_close_timeouts_total Any sustained rate > 0 The reconciler's idle-close sweep hit its 90 s per-session deadline. The session is deferred to the next tick. A steady rate > 0 indicates a session with a lease that never drops (stuck stream, hung worker, unresponsive bridge) — inspect the coincident tracing warn line for the session id.
av_shutdown_session_close_timeouts_total Any occurrence during a rollout The shutdown-time per-session close hit its 3 s deadline. The session's finalize is deferred to the daemon's next-boot spool recovery (idempotent, no receipts lost). Distinct from av_http_shutdown_drain_timeouts_total which fires on the OUTER phase timeout. Any occurrence on a rollout indicates a class of sessions that hang close — leaked SessionLease, dropped worker permit, unresponsive bridge publish. Inspect the coincident tracing warn line for the session id.
av_shutdown_mcp_drain_timeouts_total Any occurrence during a rollout The shutdown MCP-inflight drain hit its 5 s deadline before every detached mcp_call_inner spawn completed. mcp_call decouples the tool-execution body from axum's per-connection cancellation to keep durable state consistent under client disconnect; the drain waits for those bodies between HTTP graceful drain and worker.wait_idle so a late worker submission can't race the per-session close deadline. On drain timeout, shutdown still proceeds; sessions whose detached body was interrupted may be quarantined at restart-time recovery. Any occurrence on a rollout indicates a class of tool calls that regularly outlive the drain window (long-running upstream tool, hung sandbox). Distinct from av_shutdown_session_close_timeouts_total (per-session close deadline) and av_http_shutdown_drain_timeouts_total (outer HTTP-drain phase).
av_lifecycle_outbox_pending (gauge) Sustained > 0, or growth rate Count of unacked lifecycle outbox files (RECEIPT + SESSION_CLOSE) sitting in the spool at the last reconciler tick. Steady-state 0 on a healthy node. A rising baseline is the earliest signal of a broker outage — outboxes persist across restarts, so the disk-bounded queue grows until the bridge is reachable again. Investigate the bridge/broker health when this alerts.
av_recovery_scan_capped_total{pass="…"} Any sustained rate > 0 A recovery pass hit its per-tick entry-scan cap (10 000 entries) and returned early. The next tick re-opens the directory and resumes. A steady rate > 0 indicates a POISONED SPOOL (millions of stale files) that will starve legitimate recovery work until an operator quarantines the offending files. The pass label identifies which pass hit the cap: adopt_strict_atif, recover_signed_journals, consolidate_step_journals, retry_marked_promotions, remove_acked_outboxes, replay_lifecycle_outboxes, or quarantine_orphan_json.

Backup

You need three things to reconstruct evidence for an audit:

  1. The signer seed file (signer.seed) — filesystem, offline, GPG.
  2. The spool dir (atif_spool_dir) — all .atif-auth sidecars must be preserved; the corresponding .json trajectories are recoverable from the broker if lost.
  3. The receipt topic (agent.receipt on the broker) — either the embedded Bridge's data-dir or the external broker's log retention.

Losing (1) permanently invalidates every past receipt: they still verify against the historical public key, but you can no longer prove that key was yours. Losing (2) sacrifices operational forensics but does NOT compromise the cryptographic chain. Losing (3) means downstream consumers depend on the harness's live disk copy — restore quickly.

Upgrade

The workspace's SemVer contract:

  • Signed receipts: canonical (JCS + Ed25519) format is frozen. Any change is a MAJOR version bump.
  • ATIF trajectories: v1.7 shape is frozen. New fields are added as optional; removals or renames are MAJOR bumps.
  • Config: new keys added with #[serde(default)] and documented in docs/reference/CONFIGURATION.md. Removals are MAJOR bumps.
  • avctl CLI surface: adding subcommands is minor; removing them or renaming flags is MAJOR.

Rolling upgrades within a MAJOR version are safe: mix old and new pods behind the same LB and the audit chain remains verifiable under any pinned key that either version signed with.