Skip to content

Commit e46cb68

Browse files
mchwangclaude
andauthored
Enforce a pre-spawn model budget for planning invocations (#163) (#196)
* Enforce a pre-spawn model budget for planning invocations (#163) createClaudeCommand now refuses a planning request whose prompt and schema together exceed 64 KiB of UTF-8 (a byte count standing in for tokens, as Claude's tokenizer is not available offline), and passes --max-turns 50 so the run itself is bounded by turns, not only by time and output bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Quote Claude's errors when a run stops without a result (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the turn limit in planning errors; label budget assumptions (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check the turn-limit envelope against a real Claude Code 2.1.281 run (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rewrap parseClaudeOutput's comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rewrap parseClaudeOutput's comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Use the observed turn-limit envelope verbatim; note the limit-plus-one count (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Word the model-budget docs to their evidence (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tie the planning input budget to the author's limits; note text-based turn-limit detection (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rewrap the plan-author builder note Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check a prepared draft at the prompt limit against the model budget (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Split the over-budget size into prompt and schema (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name every part of the request in the over-budget advice (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say which part of the backstop claim a test covers (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rewrap the budget note Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Match the turn-limit text anywhere; note the errors text reaches every phase (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Make the other-phases note its own bullet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say that other output with the turn-limit text gets the same advice (#163) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Re-record planning fixtures with the --max-turns budget (#163) Recorded by the maintainer with scripts/record-planning.ts through the planning container (Claude Code 2.1.281). Both answers validate and replay through E2/E3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 114d962 commit e46cb68

12 files changed

Lines changed: 169 additions & 31 deletions

File tree

‎agents/adapters/claude.ts‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -47,26 +47,30 @@ function topLevelMembers(json: string): Map<string, string> {
4747

4848
/**
4949
* Read Claude's `--output-format json` envelope. A result whose `subtype` is not `success`, or whose `is_error` is
50-
* true, is a provider failure; it can omit `result`. With `--json-schema` (planning), a successful answer needs
51-
* `subtype: "success"` and is the schema-validated `structured_output` object. It is returned as the exact JSON text
52-
* of that member in the CLI's envelope, never re-encoded here, so the plan parser sees any duplicate key or inexact
53-
* integer the envelope holds (#166). Whether the API or CLI resolves those before writing the envelope is not verified.
54-
* With a schema, an envelope that repeats a top-level key is malformed, whatever it reports. `result` then holds only
55-
* Claude's prose. Plain-text envelopes without a `subtype` are still accepted.
50+
* true, is a provider failure; it can omit `result`, and when `result` is missing or empty its `errors` strings are
51+
* the text. With `--json-schema` (planning), a successful answer needs `subtype: "success"` and is the
52+
* schema-validated `structured_output` object. It is returned as the exact JSON text of that member in the CLI's
53+
* envelope, never re-encoded here, so the plan parser sees any duplicate key or inexact integer the envelope holds
54+
* (#166). Whether the API or CLI resolves those before writing the envelope is not verified. With a schema, an
55+
* envelope that repeats a top-level key is malformed, whatever it reports. `result` then holds only Claude's prose.
56+
* Plain-text envelopes without a `subtype` are still accepted.
5657
*/
5758
export function parseClaudeOutput(raw: Buffer, structured = false): { text: string; providerFailed: boolean } {
5859
const source = new TextDecoder('utf-8', { fatal: true }).decode(raw);
5960
const envelope = JSON.parse(source) as
60-
{ subtype?: unknown; result?: unknown; is_error?: unknown; structured_output?: unknown };
61+
{ subtype?: unknown; result?: unknown; is_error?: unknown; structured_output?: unknown; errors?: unknown };
6162
// Read every member before classifying: JSON.parse keeps only the last of a repeated key, such as `is_error`.
6263
const members = structured && envelope !== null && typeof envelope === 'object' && !Array.isArray(envelope)
6364
? topLevelMembers(source) : null;
6465
if (typeof envelope.is_error !== 'boolean' || (envelope.result !== undefined && typeof envelope.result !== 'string')
6566
|| (envelope.subtype !== undefined && typeof envelope.subtype !== 'string'))
6667
throw new Error('Claude returned a malformed output envelope.');
6768
const prose = envelope.result ?? '';
68-
if (envelope.is_error || (envelope.subtype !== undefined && envelope.subtype !== 'success'))
69-
return Object.freeze({ text: prose, providerFailed: true });
69+
if (envelope.is_error || (envelope.subtype !== undefined && envelope.subtype !== 'success')) {
70+
// A run the CLI stops itself, such as at `--max-turns` (#163), gives no `result`, only its `errors` strings.
71+
const errors = Array.isArray(envelope.errors) ? envelope.errors.filter(error => typeof error === 'string') : [];
72+
return Object.freeze({ text: envelope.result || errors.join('; '), providerFailed: true });
73+
}
7074
if (!structured) {
7175
if (envelope.result === undefined) throw new Error('Claude returned a malformed output envelope.');
7276
return Object.freeze({ text: prose, providerFailed: false });

‎agents/policy.ts‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,21 @@ export function dispatchApprovedCommand<T>(policy: PhasePolicy, argv: readonly s
7676
return execute(Object.freeze([...argv]));
7777
}
7878

79+
/**
80+
* Planning's model budget (#163), checked when the command is built, before the adapter creates a container. The input
81+
* is counted in UTF-8 bytes of the prompt and schema: Claude's tokenizer is not available offline, and a count of bytes
82+
* bounds the tokens if every token covers at least one byte (assumed, not verified against Claude's tokenizer). 64 KiB
83+
* is the 32 KiB prompt plus the 32 KiB of schema text that core/planning-author.ts allows, so this is a backstop for
84+
* callers that skip it; it does not count Claude Code's own system prompt and tools, or the answer. The run is bounded
85+
* by turns: if each turn sends at most one context window (assumed; the CLI's own calls, such as compaction, are not
86+
* counted), `--max-turns` bounds the tokens a run can use (the CLI may make one call more than the limit). Claude stops
87+
* at the limit with an `error_max_turns` result and exit status 1 (seen in a run of Claude Code 2.1.281 on macOS,
88+
* outside the container), which the adapter reports as a provider failure. See docs/implementation/agent-isolation.md,
89+
* "Planning model budget".
90+
*/
91+
export const MAX_PLANNING_INPUT_BYTES = 64 * 1024;
92+
export const PLANNING_MAX_TURNS = 50;
93+
7994
/**
8095
* Planning answers must match the request's schema, so the planning command carries it. Claude's `--json-schema` takes
8196
* the schema text, not a path; the profile checks that text against the mounted file. Questions answer in plain text
@@ -86,11 +101,12 @@ export function createClaudeCommand(policy: PhasePolicy, prompt: string, schema?
86101
if (assertPhasePolicy(policy).vendor !== 'claude') throw new Error('Claude command requires a Claude invocation policy.');
87102
if ((policy.phase === 'planning') !== (schema !== undefined))
88103
throw new Error('Only the Claude planning command carries an answer schema, and it must.');
89-
const schemaArguments = schema === undefined ? [] : ['--json-schema', assertPlanningSchema(schema)];
104+
const planningArguments = schema === undefined ? [] : ['--json-schema',
105+
assertPlanningInput(prompt, assertPlanningSchema(schema)), '--max-turns', String(PLANNING_MAX_TURNS)];
90106
const writable = policy.worktree === 'read-write';
91107
const allowed = writable ? 'Read,Glob,Grep,Edit,Write' : 'Read,Glob,Grep';
92108
// `--` ends option parsing, so a prompt beginning with `-` stays prompt data.
93-
return command(policy, ['claude', '--print', '--output-format', 'json', ...schemaArguments, '--restricted',
109+
return command(policy, ['claude', '--print', '--output-format', 'json', ...planningArguments, '--restricted',
94110
'--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}', '--disable-slash-commands', '--no-chrome',
95111
'--permission-prompts', 'none', '--permission-mode', writable ? 'acceptEdits' : 'plan', '--tools', allowed,
96112
'--allowedTools', allowed, '--disallowedTools', 'Bash,WebFetch,WebSearch,NotebookEdit', '--add-dir',
@@ -117,6 +133,16 @@ export function createRunnerCommand(policy: PhasePolicy, commands: string): Agen
117133
return command(policy, ['node', '/usr/local/bin/codeboost-command-check', '/run/codeboost-input/schema.json'], commands);
118134
}
119135

136+
/** Refuse a planning request whose prompt and schema together exceed the model budget. Returns the schema unchanged. */
137+
export function assertPlanningInput(prompt: string, schema: string): string {
138+
const promptBytes = Buffer.byteLength(prompt, 'utf8'), schemaBytes = Buffer.byteLength(schema, 'utf8');
139+
if (promptBytes + schemaBytes > MAX_PLANNING_INPUT_BYTES)
140+
throw new Error(`Planning request is ${promptBytes + schemaBytes} bytes (prompt ${promptBytes}, schema ${schemaBytes}), `
141+
+ `over the ${MAX_PLANNING_INPUT_BYTES}-byte model budget. Narrow the request (the issue and comments, the selected `
142+
+ 'paths, the lessons, the feedback or the previous plan), then retry.');
143+
return schema;
144+
}
145+
120146
/**
121147
* A planning schema must describe a JSON object (Claude returns `structured_output` as an object) and fit one command
122148
* argument. Returns the schema unchanged.

‎docs/implementation/agent-isolation.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ Each row is a T9 requirement for the Docker suite. The suite fails if any row fa
3636
| A read-only phase cannot write `/work` | `agent-container`: phase worktree for all five phases |
3737
| Planning and questions cannot run a process | `agent-policy`: tool sets exclude the command tool, and command dispatch refuses these phases |
3838
| Codex runs in no phase, because it cannot read code without its shell (see [Codex is refused in every phase](#codex-is-refused-in-every-phase)) | `agent-policy`: the Codex command refuses all five phases; `agent-adapter`: the Codex start call refuses them and allocates nothing; `runner-execution`: an execute task with Codex is refused before task storage exists |
39+
| A planning request over its model budget is refused before the start, and the planning command passes `--max-turns 50` (see [planning model budget](#planning-model-budget) for what is and is not shown about the stop) | `agent-policy`: a prompt and schema of exactly 64 KiB build a command with `--max-turns 50`, and one byte more is refused; `agent-adapter`: the start call refuses an over-budget request and allocates nothing, and an `error_max_turns` envelope is a provider failure whose text is its `errors`; `planning-provider`: the planning error quotes the turn limit |
3940
| A planning answer is checked against the mounted schema | `agent-policy`: only the Claude planning command passes `--json-schema`; `agent-container`: the profile refuses a command whose schema differs from the mounted file; `agent-adapter`: the answer is read from `structured_output` |
4041
| The Docker daemon keeps no log of an agent container (#165): the supervisor's bounded capture through `docker start --attach` is the only copy of its output. This covers the agent container only. The runner's other containers keep the daemon's default log: the task-storage script containers (`docker run --rm`, `agents/container/storage.ts`), whose log is removed with the container while the client bounds what it reads, and the storage keeper and egress proxy, which write no output | `agent-container`: the created container's `LogConfig` is driver `none` with no option but `cache-*` ones; the validator refuses a container with a `json-file` log or a bounded `local` log; every probe that returns output shows the attach still delivers it. `agent-daemon-log`: the rule (`keepsNoDaemonLog`) also refuses driver `none` with a log option, which only the API can set, and accepts the `cache-*` options a daemon copies into every container |
4142
| Task and scratch byte and inode limits hold | `agent-container`: task capacity; scratch capacity for Codex and Claude (`/tmp`, `HOME`, `CODEX_HOME`, output directory) |
@@ -102,6 +103,36 @@ Use only these entry points to run an agent:
102103
one link, at most 64 KiB of UTF-8, and a JSON object with `"type": "object"`; otherwise the start call throws.
103104
Leave out a draft 2020-12 `$schema` line: Claude's `--json-schema` rejects it (see [plan
104105
format](../plan-format.md)).
106+
- <a id="planning-model-budget"></a>**Planning model budget (#163).** `createClaudeCommand` checks it when it
107+
builds the planning command, so the adapter's start call throws before it creates a container or network. The
108+
runner's host clone and task storage already exist by then, and the runner removes them. The budget has two parts:
109+
- *Input, before the start.* The prompt and the schema together must be at most `MAX_PLANNING_INPUT_BYTES` (64 KiB)
110+
of UTF-8: the 32 KiB prompt plus 32 KiB of schema text. A larger request is refused with an error that gives its
111+
size, split into prompt and schema, and asks the person to narrow the request (the issue and comments, the paths,
112+
the lessons, the feedback or the previous plan). Bytes stand in for tokens because Claude's tokenizer is not
113+
available offline. A count of bytes is an upper bound on the tokens if every token covers at least one byte; this
114+
is assumed, not verified against Claude's tokenizer. `prepareDraft` and `prepareSuggestions` already limit the
115+
prompt and the schema text to 32 KiB each, so this check refuses nothing they produce. It is a backstop for a
116+
caller that skips them. A test checks a draft at the prompt limit with today's schema; no test fails if the
117+
32 KiB schema bound in `core/planning-author.ts` is removed. It does not count Claude Code's own system prompt
118+
and tool definitions, or the tokens of the answer.
119+
- *The run.* The command passes `--max-turns 50` (`PLANNING_MAX_TURNS`). If each turn sends at most one context
120+
window, a turn limit bounds the tokens a run can use with no tokenizer. That is assumed: calls the CLI makes for
121+
itself, such as compaction, may not count as turns, and the bound is loose (about 50 context windows; the CLI
122+
reported `num_turns: 2` when it stopped at a limit of 1, so it may make one call more than the limit). Claude
123+
Code 2.1.281 accepts the flag in `--print` mode (its help hides it) and stops at the limit with an
124+
`error_max_turns` result and exit status 1. This was checked with `--max-turns 1` in a run of the 2.1.281 macOS
125+
arm64 build outside the container; the Linux build in the image was read from its bundled source only. The
126+
adapter reports that result as a provider failure whose text is the result's `errors` strings ("Reached maximum
127+
number of turns (1)" in that run). That holds in every Claude phase, so a failed question run quotes them too.
128+
The planning error then names the 50-turn limit, asks the person to narrow the request, and quotes that text. The
129+
planning error finds the stop by that text, so other output that contains it, such as Claude's own `result`, gets
130+
the same advice; the output is quoted either way. A token or dollar limit was not chosen: `--max-budget-usd`
131+
counts the dollar cost of API calls, which is not what an OAuth subscription token pays, and the CLI has no flag
132+
that limits the tokens of a whole run. 50 turns is about 12 s for each turn in the ten-minute budget; a planning
133+
run reads files and answers, so most turns are reads. Lower it if runs are seen to loop; raise it only with a
134+
recorded run that needs more.
135+
- *Other phases.* They do not pass `--max-turns` yet; their runs are bounded by time and output bytes only.
105136
- **Codex.** `startCodexInvocation` throws for every phase, and allocates nothing. Use Claude. See
106137
[Codex is refused in every phase](#codex-is-refused-in-every-phase).
107138
5. To keep a stopped writable attempt's partial output, call `exportTaskDiff(storage, { base, imageId })` before

‎docs/implementation/plan-library-alignment.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ Issue #6 tracked the v1 plan library contract and the runner obligations that ca
2323
| Output shown in the UI bounded: captured output comes only from the supervisor, which bounds stdout and stderr; when a planning run exits with an error, the provider's message quotes its output with whitespace collapsed, cut to at most 300 UTF-16 code units (`runner/planning-provider.ts`) | Done before this change; 300-unit quote tested in this change | `test/agent-supervisor.test.ts` output-limit cases (real Docker); `test/planning-provider.test.ts` ("quotes at most 300 code units of a failed run's output, with whitespace collapsed"). The Docker daemon's own log is #165 (*later:* the daemon keeps none) |
2424
| Output limits at the exact byte | **This change** (tests; the supervisor already enforced the limits) | Supervisor enforcement: `test/agent-supervisor.test.ts` (real Docker). Decoder behaviour only, not a limit: the Claude decoder handles an envelope of exactly the stdout ceiling, and the 1 MiB parser limit applies to the decoded answer at the exact byte: `test/agent-adapter.test.ts`. At #6 the decoder re-serialized the answer, so the limit applied to the re-encoded text. *Later:* #166 returns the exact text of `structured_output` in the CLI's envelope, so the limit applies to those bytes |
2525
| Prompt byte budget before spawn | Done before this change | `MAX_PROMPT_BYTES` in `core/planning-author.ts`; `test/planning-author.test.ts` ("accepts exactly the prompt byte limit and rejects the next byte without truncation") |
26-
| Model budget before spawn | Moved to #163 | — |
26+
| Model budget before spawn | Moved to #163. *Later:* done in #163 | `MAX_PLANNING_INPUT_BYTES` and `PLANNING_MAX_TURNS` in `agents/policy.ts`; `test/agent-policy.test.ts` ("accepts a planning request at exactly its model budget and refuses one byte more"); [agent isolation](agent-isolation.md#planning-model-budget) |
2727
| Planning profile probed at startup | Moved to #164 | — |
2828
| Docker daemon log limits | Moved to #165. *Later:* fixed in #165 with `--log-driver=none`, checked before launch | `agents/container/profile.ts`, `agents/container/run.ts`; `test/agent-container.test.ts` ("creates the agent container with no Docker daemon log", "refuses an agent container with %s at validation") (real Docker), `test/agent-daemon-log.test.ts`. Agent containers only; the runner's other containers (task-storage scripts, the keeper, the egress proxy) keep the daemon's default log |
2929
| Claude structured output under the v1 parsing contract | Moved to #166 (found during this change). *Later:* fixed in #166 | `agents/adapters/claude.ts`; `test/agent-adapter.test.ts` ("lets draft validation refuse %s in a structured answer") |

‎prompts/plan-author.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -40,12 +40,14 @@
4040
Reject NUL input and any oversized request before spawning; do not silently
4141
truncate any source. Report which input exceeded the budget and ask the person
4242
to narrow the issue/comments, selected path set, or lesson set before retrying.
43-
Also enforce the selected pinned model profile's token budget (including
44-
schema/system overhead and reserved answer tokens) before launch; missing
45-
budget/tokenizer support fails closed. The launch probe must test these caps
46-
on the supported container OS/CLI; lower them if needed, never raise them
47-
automatically. Tests include one huge field, many small fields exceeding the
48-
aggregate, post-escaping expansion, and exact boundary accepted/rejected inputs.
43+
Also enforce the model budget before launch. With no tokenizer available
44+
offline, lane D counts the prompt and schema in UTF-8 bytes (64 KiB together)
45+
and bounds the run with Claude's --max-turns (#163; see agent-isolation.md,
46+
"Planning model budget"; no probe tests them yet, #164). The launch probe
47+
must test these caps on the supported container OS/CLI; lower them if
48+
needed, never raise them automatically. Tests include one huge field, many
49+
small fields exceeding the aggregate, post-escaping expansion, and exact
50+
boundary accepted/rejected inputs.
4951
In every serialized data string, escape <, >, and & as JSON Unicode escapes. Never insert
5052
raw source text or recursively render placeholders inside serialized values.
5153
These wrappers do not prevent semantic prompt injection: container permissions,

‎runner/planning-provider.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import { tmpdir } from 'node:os';
33
import { basename, dirname, join } from 'node:path';
44
import type { InvocationInput, InvocationResult, StopReason } from '../agents/contract.ts';
55
import type { TaskStorageLimits } from '../agents/container/storage.ts';
6+
import { PLANNING_MAX_TURNS } from '../agents/policy.ts';
67
import type { AuthorProvider, AuthorRequest } from '../core/planning-author.ts';
78
import { allocations, exceedsStorage, RetainedStorage, runReadOnlyAgent, sameContext, type ContainerDependencies, type LeftoverPolicy, type Provider,
89
type ReadOnlyFeature, type ReadOnlyRun, type RepositorySize } from './question-container.ts';
@@ -94,7 +95,13 @@ export function planningOutput(result: InvocationResult, invocation: InvocationI
9495
throw new Error(`Claude stopped unexpectedly${result.signal ? ` (${result.signal})` : ''}.`);
9596
if (result.exitCode !== 0) {
9697
const detail = result.stdout.replace(/\s+/g, ' ').trim().slice(0, 300);
97-
throw new Error(`Claude could not write the plan. Check its sign-in and usage limits.${detail ? ` Claude said: ${detail}` : ''}`);
98+
// Claude Code's own text when `--max-turns` stops the run (#163). The decoder keeps no subtype, so this matches the
99+
// text: a stop without it gets the general advice, and other output that contains it (Claude's own `result` text
100+
// can) gets the turn-limit advice. Only the advice can be wrong; the text is still quoted.
101+
const advice = detail.includes('Reached maximum number of turns')
102+
? `Planning stopped at its ${PLANNING_MAX_TURNS}-turn limit before Claude answered. Narrow the request, then retry.`
103+
: 'Check its sign-in and usage limits.';
104+
throw new Error(`Claude could not write the plan. ${advice}${detail ? ` Claude said: ${detail}` : ''}`);
98105
}
99106
return result.stdout;
100107
}

0 commit comments

Comments
 (0)