From dfd0012418997faf6cb46775b3b48ed74930a06f Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Thu, 30 Jul 2026 14:54:01 -0400 Subject: [PATCH 1/3] Add `agent-run-cost` skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scripted automation announces its cost in wall-clock time; agentic automation does not. A fan-out of forty subagents and a single call are the same shape in a diff, and the difference surfaces later on a bill attributed to nothing in particular. The token-spend counterpart to `sentry-quota`, and deliberately the same posture: operate on code and PRs before the spend exists, and produce figures rather than adjectives. Names the amplifier triad — fan-out × trigger frequency × no-kill-switch — where one alone is usually fine and all three together is the shape that produces a surprise. Requires the arithmetic be shown, and the worst case stated separately from the expected case, since the budget conversation is about the PR that touches 400 files rather than the normal one. Renders no ship verdict: whether a cost is worth paying belongs to whoever owns the budget. Raised as an open question during ADR-0058 review (MetaMask/decisions#173), where agent token consumption had no estimate. --- .../agentic/skills/agent-run-cost/skill.md | 108 ++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 domains/agentic/skills/agent-run-cost/skill.md diff --git a/domains/agentic/skills/agent-run-cost/skill.md b/domains/agentic/skills/agent-run-cost/skill.md new file mode 100644 index 00000000..f44ee8b0 --- /dev/null +++ b/domains/agentic/skills/agent-run-cost/skill.md @@ -0,0 +1,108 @@ +--- +name: agent-run-cost +maturity: experimental +description: >- + Estimate what an agentic workflow costs to run before it merges — fan-out × trigger + frequency × no kill-switch — and say so in figures rather than adjectives. Agent token + spend is invisible in a diff: a workflow that spawns one agent and one that spawns forty + are the same few lines, and the difference only appears on a bill nobody reads during + review. Produces a per-run and per-month estimate with its arithmetic shown, flags the + three amplifiers, and proposes the cheapest mitigation that preserves the intent. Use + when a PR adds or widens an agentic workflow, an AEP task class, a verification recipe, + or a schedule that runs agents unattended. +--- + +# Agent run cost + +Scripted automation announces its cost in wall-clock time; agentic automation does not. +A fan-out of forty subagents and a single call are the same shape in a diff, and the +difference surfaces later, on a bill, attributed to nothing in particular. + +This is the token-spend counterpart to `sentry-quota`, which guards span volume. Same +posture: operate on **code and PRs**, before the spend exists, and produce figures. + +## When to use + +- A PR adds or widens an agentic workflow, AEP task class, or verification recipe. +- A workflow gains fan-out — an agent per file, per finding, per test, per PR. +- Something agentic moves from opt-in to automatic (a CI trigger, a cron, a git hook). +- An ADR or design proposes agents for work a script already does — the estimate is the + argument, and its absence is usually the tell. + +## Do not use when + +- The workflow is developer-invoked, one agent, no loop — the ceiling is a person's patience. +- The change only narrows fan-out or adds a gate. + +## The amplifier triad + +Cost is not the per-agent price. It is the product of three things, any one of which can be +the whole problem: + +| Amplifier | What it looks like | Effect | +|---|---|---| +| **Fan-out** | an agent per item — per file, per finding, per dimension, per round; nested `parallel` inside `pipeline`; a loop-until-dry with no ceiling | N× per run, and N is often data-dependent rather than fixed | +| **Trigger frequency** | runs on every push rather than on demand; a cron; a label that re-fires on each commit; a retry that respawns the fleet | turns a one-off into a rate | +| **No kill-switch** | no env var, feature flag, or budget cap; nothing to stop it mid-run; no way to disable without a revert | a runaway costs whatever it costs until someone merges a fix | + +One alone is usually fine. **Fan-out × frequency with no kill-switch is the shape that +produces a surprise**, and it is worth naming explicitly in review when all three are present. + +## Producing the estimate + +Show the arithmetic. An estimate whose derivation is hidden is an adjective. + +1. **Count agents per run.** Read the fan-out literally — how many items feed the widest + stage, and whether that number is bounded by the code or by the data. A `pipeline` over + changed files is unbounded by the code; `Array.from({length: 3})` is not. +2. **Estimate tokens per agent.** Prompt + the context it will read + its output. The context + dominates: an agent that reads three files is not an agent that greps a repo. +3. **Multiply, then apply frequency.** Per-run cost × runs per week. State the assumption + about run count — it is the number most likely to be wrong, and naming it lets a reviewer + correct it. +4. **State the worst case separately from the expected case.** The expected case is what it + costs on a normal PR; the worst case is what it costs on the PR that touches 400 files. + Budget conversations are about the second one. +5. **Compare against the alternative.** If a deterministic script covers the same ground, the + estimate belongs next to that script's cost. An agentic approach can still win — for + adversarial review, exploration, fuzzing, or anything with no fixed oracle — but the case + is made by the comparison, not by the capability. + +Report the figures, the assumptions behind them, and the mitigation. **Do not render a +ship/no-ship verdict** — whether a cost is worth paying is a budget decision, and it belongs +to whoever owns the budget. + +## Mitigation ladder + +Cheapest first; stop at the rung that fits. + +1. **Cap the fan-out.** A literal ceiling on items, with a `log()` of what was dropped — + silent truncation reads as full coverage and is worse than the cost. +2. **Narrow the trigger.** On-demand or label-gated instead of every push; on the changed + subset instead of the tree. +3. **Right-size the model per stage.** Mechanical stages rarely need the top tier; reserve it + for the judgement stages. +4. **Add a budget guard.** A token ceiling the workflow checks between stages, so it degrades + instead of running to completion at any price. +5. **Add a kill-switch.** An env var or flag that disables it without a revert. Cheap to add + up front and unavailable exactly when it is needed most. + +## Common pitfalls + +| Mistake | Correct approach | +|---|---| +| "It's just a few agents" | Count them. Data-dependent fan-out has no "just" | +| Estimating output tokens only | Context dominates — an agent that reads the repo costs more than one that answers at length | +| Quoting an average with no worst case | The worst case is the budget conversation | +| Treating a retry as free | A retried fleet is a second fleet | +| Assuming a concurrency cap bounds cost | It bounds *parallelism*, not total spend — queued agents still run | +| Adding a kill-switch after launch | It is needed during the incident it would have prevented | +| Comparing capability instead of cost | "Agents can do this" is not "agents should do this at this price" | + +## Related + +- `sentry-quota` — the same guard for span volume; `fan-out × ungated × no-kill-switch`. +- `pr-validate` — weighs AEP run cost when choosing an evidence lane, and tears the stack + down after; this skill is the review-side version for workflows others will run. +- [`MetaMask/decisions#173`](https://github.com/MetaMask/decisions/pull/173) — ADR-0058 + review, where the missing token-cost estimate was raised as an open question. From 698d4f10df7820937793a2e2632d7acbf6211d27 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Fri, 31 Jul 2026 10:26:58 -0400 Subject: [PATCH 2/3] Drop references to a skill that does not exist from `agent-run-cost` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `sentry-quota` was named twice — once as the framing analogy, once in `## Related` — but it exists on no branch of this repo. The analogy is kept as a description of the guard; the `## Related` entry is removed. Also updates `pr-validate` to its current name, `evidence`. --- domains/agentic/skills/agent-run-cost/skill.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/domains/agentic/skills/agent-run-cost/skill.md b/domains/agentic/skills/agent-run-cost/skill.md index f44ee8b0..b37dbc30 100644 --- a/domains/agentic/skills/agent-run-cost/skill.md +++ b/domains/agentic/skills/agent-run-cost/skill.md @@ -18,7 +18,7 @@ Scripted automation announces its cost in wall-clock time; agentic automation do A fan-out of forty subagents and a single call are the same shape in a diff, and the difference surfaces later, on a bill, attributed to nothing in particular. -This is the token-spend counterpart to `sentry-quota`, which guards span volume. Same +This is the token-spend counterpart to a span-volume quota guard. Same posture: operate on **code and PRs**, before the spend exists, and produce figures. ## When to use @@ -101,8 +101,7 @@ Cheapest first; stop at the rung that fits. ## Related -- `sentry-quota` — the same guard for span volume; `fan-out × ungated × no-kill-switch`. -- `pr-validate` — weighs AEP run cost when choosing an evidence lane, and tears the stack +- `evidence` — weighs AEP run cost when choosing an evidence lane, and tears the stack down after; this skill is the review-side version for workflows others will run. - [`MetaMask/decisions#173`](https://github.com/MetaMask/decisions/pull/173) — ADR-0058 review, where the missing token-cost estimate was raised as an open question. From d04bd59932ab5ce10b99a98247b57c83cd320468 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Fri, 31 Jul 2026 12:34:08 -0400 Subject: [PATCH 3/3] Restore the `sentry-quota` references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removed in the previous commit on the claim that no such skill existed. It does — `sentry-quota` ships in the analytics domain. The check that "proved" its absence passed a bare PR number where a ref was required, so every lookup errored into a silenced zero and the skill appeared to exist nowhere. A skill defined in a concurrent pull request is a forward reference that resolves on merge, which is why the reference linter treats it as a warning rather than an error. --- domains/agentic/skills/agent-run-cost/skill.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/domains/agentic/skills/agent-run-cost/skill.md b/domains/agentic/skills/agent-run-cost/skill.md index b37dbc30..7feb8060 100644 --- a/domains/agentic/skills/agent-run-cost/skill.md +++ b/domains/agentic/skills/agent-run-cost/skill.md @@ -18,7 +18,7 @@ Scripted automation announces its cost in wall-clock time; agentic automation do A fan-out of forty subagents and a single call are the same shape in a diff, and the difference surfaces later, on a bill, attributed to nothing in particular. -This is the token-spend counterpart to a span-volume quota guard. Same +This is the token-spend counterpart to `sentry-quota`, which guards span volume. Same posture: operate on **code and PRs**, before the spend exists, and produce figures. ## When to use @@ -101,6 +101,7 @@ Cheapest first; stop at the rung that fits. ## Related +- `sentry-quota` — the same guard for span volume; `fan-out × ungated × no-kill-switch`. - `evidence` — weighs AEP run cost when choosing an evidence lane, and tears the stack down after; this skill is the review-side version for workflows others will run. - [`MetaMask/decisions#173`](https://github.com/MetaMask/decisions/pull/173) — ADR-0058