Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions docs/research/CHILD_ACQUISITION_AND_PROGRESS_2026-09-07.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Child acquisition and progress reliability audit

- Status: validated for deterministic defect reproduction; repair acceptance is recorded in the linked PR
- Created / verified: 2026-09-07
- Source boundary: `eaf470bab4ac2dda607d16c6ddee66dc35367527` (main after PR #426)
- Issue: [#428](https://github.com/openpi-dev/openpi/issues/428)
- Repair PR: [#429](https://github.com/openpi-dev/openpi/pull/429)
- Supersedes: none

## Scope and provenance

This audit checks child startup ownership and progress projection. The repair uses an isolated source worktree; `pi list` still identifies the user's separate `openpi-main-runtime` checkout at `c8f2c13`. Source validation does not imply that the user's already-running Pi loaded the fixes. No user role files, settings, private Sessions, or ignored evidence were changed.

Existing viewport, retained-history, headless-shell and terminal-artifact PRs (#327, #319, #423 and #386) remain separate. The previous Cursor/tool/cwd repair was merged as [#426](https://github.com/openpi-dev/openpi/pull/426).

## Confirmed mechanisms

### Cancelled ordinary child acquisition loses its owner

The production Pi backend used an interruptible `Effect.tryPromise` acquisition but ignored its abort signal. Its cleanup finalizer was registered only after resource loading, session creation and extension binding. A deterministic probe paused the session factory, interrupted the scoped spawn, then released the factory. The late child bound extensions despite cancellation and received zero abort, shutdown or dispose calls.

The repair gives startup provisional cleanup ownership before the first asynchronous acquisition, observes cancellation at acquisition boundaries and reuses the bounded child shutdown helper. A child arriving after cancellation is disposed without beginning extension binding. Ownership transfers to the existing backend finalizer after registration. The tool boundary preserves interruption as a typed error, including combined interruption and cleanup failure, instead of treating it as a known quiescent startup failure. For isolated startup, it retains the checkout and reports its path/branch while quiescence is unknown; a paused binding hook can therefore finish writing without its cwd disappearing. Known non-interrupted startup failures retain the existing empty-worktree cleanup.

Already-running extension hooks cannot be forcibly stopped by an SDK API that offers no cancellation; bounded cleanup is not a claim that arbitrary extension code becomes cancellable.

### Cancelled Workflow worktree acquisition skips cleanup

A disposable Git repository's post-checkout hook held `git worktree add` in progress. Cancelling the Workflow, releasing the hook and awaiting run completion left two worktrees instead of one. The abort check threw after successful creation but before the owning `try/finally`.

The repair moves that check inside the existing cleanup region. Existing policies remain authoritative: an empty worktree may be reclaimed, while user-produced dirty work and uncertain cleanup retain their evidence.

### Byte pressure hides the newest failure

With an initial task, twenty 16-KiB tool results and a final error, the production progress projection omitted the final error and last showed `call-15`. Its entry-count policy retained the first and newest entries, but its byte-budget pass consumed the budget from oldest to newest.

The repair reserves the initial entry and allocates the remaining text budget from newest to oldest, then restores chronological display. Tool identity, error state and timing travel with retained entries. Existing per-entry truncation and omission markers remain explicit. The 256-KiB limit is the source-text budget; markers and metadata are not a claim of a strict serialized-object byte limit. Pi's canonical messages and tool results are unchanged.

### Tool preview scans content it never displays

`toolPreview` split the entire tool output into lines for every update even though the UI needed only its first nonempty line. An exact-source synthetic probe returned the same five characters (`ready`) while ten updates over an 8-MiB dense-line payload took approximately 442 ms. This is a stress case, not a claim that ordinary Pi Bash updates routinely contain 8 MiB; native tools may already truncate their partial results.

The repair finds the first meaningful text and examines only the first-line preview window, using the existing consumer's 64-KiB character limit. Necessary leading-whitespace search remains linear; trailing multiline output no longer creates a whole-output array. Full tool results remain canonical and available independently of the preview.

One local Node 26.3.0 diagnostic comparison reused identical text payloads, extracted the before function from the frozen source and imported the repaired helper. Both returned `ready`; timing excluded construction of the payload. This isolates preview work rather than end-to-end latency:

| Payload and repeated calls | Before | After |
| --- | --- | --- |
| 50 KiB, 80-character lines, 1000 calls | 8.882 ms | 0.310 ms |
| Synthetic 8 MiB, 80-character lines, 100 calls | 165.259 ms | 0.013 ms |
| Synthetic 8 MiB, short lines, 10 calls | 440.523 ms | 0.003 ms |

Small after-values approach timer/JIT noise. The useful result is removal of whole-tail splitting, enforced by a deterministic regression, not a precise whole-application speedup multiplier.

## Evidence and limits

Regression tests exercise production Effect cancellation and the real Git worktree lifecycle, plus byte-pressure error retention and first-line preview behavior. `bun run check` passed. Standard `bun run test` passed 1446 Node tests with one platform skip and 30 Vitest tests. Two independent reviews are clean after closing the interrupted-startup worktree classification gap. Remote CI status is recorded in the PR. No paid model calls are needed to reproduce these deterministic runtime defects.

A separate synthetic Cursor transport probe identified repeated copying while accumulating a large fragmented Connect frame. It is outside this bounded child acquisition/progress repair, as are speculative Graph recomputation and renderer-retention concerns without a completed failure proof. This record is a diagnostic investigation, not a formal throughput Benchmark or an assertion that every possible performance issue was resolved.
2 changes: 2 additions & 0 deletions docs/research/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Research records preserve sourced investigation and distinguish observations, in

## Validated investigations

- [`CHILD_ACQUISITION_AND_PROGRESS_2026-09-07.md`](CHILD_ACQUISITION_AND_PROGRESS_2026-09-07.md) — cancelled child/worktree acquisitions and stale progress evidence under output pressure ([#428](https://github.com/openpi-dev/openpi/issues/428)).

- [`WORKFLOW_CHILD_FAILURES_2026-09-07.md`](WORKFLOW_CHILD_FAILURES_2026-09-07.md) — child tool transport, cwd and timeout failure mechanisms, intended capability inheritance, and acceptance limits ([#424](https://github.com/openpi-dev/openpi/issues/424)).

- [`WORKFLOW_DASHBOARD_REFRESH_2026-09-07.md`](WORKFLOW_DASHBOARD_REFRESH_2026-09-07.md) — repeated synchronous history loading on dashboard animation ticks, its regression boundary, and measurement limits ([#420](https://github.com/openpi-dev/openpi/issues/420)).
Expand Down
16 changes: 14 additions & 2 deletions extensions/subagents/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ import { createSubagentResultDelivery } from "./src/result-delivery.ts";
import {
createSubagentRuntime,
runTool,
SubagentToolInterruptedError,
type SubagentRuntime,
} from "./src/runtime.ts";
import { openSubagentPicker, openSubagentTakeover } from "./src/ui/takeover.ts";
Expand Down Expand Up @@ -985,11 +986,22 @@ export default function (
interruptMessage: "Subagent spawn aborted.",
});
} catch (error) {
// The session scope owns reclamation, but it never opened, so this
// worktree would otherwise be orphaned on disk.
// Known startup failures can reclaim their empty checkout. Interrupted
// startup must preserve it while asynchronous acquisition may continue.
if (worktree) {
const spawnError =
error instanceof Error ? error.message : String(error);
// Cancelling Effect acquisition does not prove an asynchronous
// factory or extension hook has quiesced. It may still use this cwd.
if (
signal?.aborted ||
error instanceof SubagentToolInterruptedError
) {
throw new Error(
`${spawnError}; startup quiescence is unknown; checkout preserved at ${worktree.path} (branch ${worktree.branch})`,
{ cause: error },
);
}
let cleanupWarning: string | undefined;
let cleanupError: unknown;
try {
Expand Down
129 changes: 70 additions & 59 deletions extensions/subagents/src/backends/pi.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import {
import type { Cause, Scope } from "effect";
import { Effect, Queue, Stream } from "effect";
import { resolveAgentModel } from "../agent-types.ts";
import { toolPreview } from "./tool-preview.ts";
import type {
SubagentBackend,
SubagentCleanupReceipt,
Expand Down Expand Up @@ -118,27 +119,6 @@ function safeJson(value: unknown): string | undefined {
}
}

/** First non-empty line of a tool result-ish value (v1 liveToolPreview). */
function toolPreview(value: unknown): string | undefined {
if (typeof value === "string") {
return value
.split("\n")
.find((line) => line.trim())
?.trim();
}
if (!value || typeof value !== "object") return undefined;
const content = (value as { content?: unknown }).content;
if (!Array.isArray(content)) return undefined;
for (const part of content) {
if (!part || typeof part !== "object") continue;
const record = part as { type?: unknown; text?: unknown };
if (record.type !== "text" || typeof record.text !== "string") continue;
const firstLine = record.text.split("\n").find((line) => line.trim());
if (firstLine) return firstLine.trim();
}
return undefined;
}

function assistantParts(msg: AssistantMessage): TranscriptPart[] {
const parts: TranscriptPart[] = [];
for (const part of msg.content) {
Expand Down Expand Up @@ -215,56 +195,85 @@ const makePiSession = (
capturedStructured = encodeStructuredResult(value);
});

// Own the session before asynchronous startup. Interruption can happen
// before the normal backend finalizer has been installed.
let acquiringSession: AgentSession | undefined;
let startupOwned = true;
const cleanupStartup = () =>
acquiringSession
? shutdownAndDisposeChildSession(acquiringSession, {
abort: true,
timeoutMs: options.shutdownTimeoutMs,
})
: Promise.resolve();
yield* Effect.addFinalizer(() =>
Effect.promise(async () => {
if (startupOwned) await cleanupStartup();
}),
);

const session = yield* Effect.tryPromise({
try: async () => {
const appendSystemPrompt = [
...(task.appendSystemPrompt ?? []),
...(structuredOutputTool
? [STRUCTURED_OUTPUT_SYSTEM_INSTRUCTION]
: []),
];
const { loader, settingsManager } = await createChildResources({
cwd: task.cwd,
projectTrusted: task.parent.projectTrusted,
...(appendSystemPrompt.length > 0 ? { appendSystemPrompt } : {}),
});
const { session } = await (
options.sessionFactory ?? createAgentSession
)({
cwd: task.cwd,
sessionManager: SessionManager.create(task.cwd),
settingsManager,
resourceLoader: loader,
model,
thinkingLevel,
...(structuredOutputTool
? { customTools: [structuredOutputTool] }
: {}),
...childToolPolicy(
childToolsWithStructuredOutput(
task.tools,
structuredOutputTool !== undefined,
),
),
});
// Start child extension session hooks/resources in headless mode.
// A rejection here would otherwise leak the freshly created session:
// the scope finalizer that owns cleanup is only registered later.
try: async (signal) => {
const checkCancelled = () => {
if (signal.aborted)
throw signal.reason ?? new Error("Subagent startup cancelled");
};
const onCancelled = () => {
void cleanupStartup().catch(() => {});
};
signal.addEventListener("abort", onCancelled, { once: true });
try {
checkCancelled();
const appendSystemPrompt = [
...(task.appendSystemPrompt ?? []),
...(structuredOutputTool
? [STRUCTURED_OUTPUT_SYSTEM_INSTRUCTION]
: []),
];
const { loader, settingsManager } = await createChildResources({
cwd: task.cwd,
projectTrusted: task.parent.projectTrusted,
...(appendSystemPrompt.length > 0 ? { appendSystemPrompt } : {}),
});
checkCancelled();
const { session } = await (
options.sessionFactory ?? createAgentSession
)({
cwd: task.cwd,
sessionManager: SessionManager.create(task.cwd),
settingsManager,
resourceLoader: loader,
model,
thinkingLevel,
...(structuredOutputTool
? { customTools: [structuredOutputTool] }
: {}),
...childToolPolicy(
childToolsWithStructuredOutput(
task.tools,
structuredOutputTool !== undefined,
),
),
});
acquiringSession = session;
checkCancelled();
// Never start extension binding for a factory that completed after
// cancellation. Already-running hooks retain bounded cleanup ownership.
await bindChildSessionExtensions(
session,
childToolsWithStructuredOutput(
task.tools,
structuredOutputTool !== undefined,
),
);
checkCancelled();
return session;
} catch (error) {
await shutdownAndDisposeChildSession(session, {
timeoutMs: options.shutdownTimeoutMs,
});
await cleanupStartup();
throw error;
} finally {
signal.removeEventListener("abort", onCancelled);
}
return session;
},
catch: (error) => new SpawnError({ message: boundedError(error) }),
});
Expand Down Expand Up @@ -650,6 +659,8 @@ const makePiSession = (
}),
);

startupOwned = false;

/** Start a fresh run (v1 manager.run): fire-and-forget, errors -> events. */
const startRun = (text: string) => {
if (state.activePrompt) {
Expand Down
29 changes: 29 additions & 0 deletions extensions/subagents/src/backends/tool-preview.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
// Match the manager's existing transcript text limit; canonical Pi tool results
// remain intact. Only the normalized event's single-line preview is bounded.
const TOOL_PREVIEW_MAX_LENGTH = 64 * 1_024;

function firstMeaningfulLine(text: string) {
// Search only until the first non-whitespace character. Unlike splitting the
// entire log, this preserves blank-line behavior without visiting its tail.
const start = text.search(/\S/);
if (start < 0) return undefined;
const prefix = text.slice(start, start + TOOL_PREVIEW_MAX_LENGTH);
const newline = prefix.indexOf("\n");
return (newline < 0 ? prefix : prefix.slice(0, newline)).trimEnd();
}

/** First meaningful line of a tool result, without splitting accumulated logs. */
export function toolPreview(value: unknown) {
if (typeof value === "string") return firstMeaningfulLine(value);
if (!value || typeof value !== "object") return undefined;
const content = (value as { content?: unknown }).content;
if (!Array.isArray(content)) return undefined;
for (const part of content) {
if (!part || typeof part !== "object") continue;
const record = part as { type?: unknown; text?: unknown };
if (record.type !== "text" || typeof record.text !== "string") continue;
const firstLine = firstMeaningfulLine(record.text);
if (firstLine) return firstLine;
}
return undefined;
}
13 changes: 10 additions & 3 deletions extensions/subagents/src/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ export function createSubagentRuntime(config: SubagentManagerConfig = {}) {

export type SubagentRuntime = ReturnType<typeof createSubagentRuntime>;

/** Canonical interruption, distinct from a known startup failure. */
export class SubagentToolInterruptedError extends Error {}

/**
* Run an effect from an async tool handler. Typed failures and defects are
* converted to thrown Errors (what pi's tool contract expects); interruption
Expand All @@ -60,9 +63,13 @@ export async function runTool<A, E>(
options.signal ? { signal: options.signal } : undefined,
);
if (Exit.isSuccess(exit)) return exit.value;
if (Cause.hasInterruptsOnly(exit.cause)) {
throw new Error(options.interruptMessage ?? "Operation was aborted.");
}
const [first] = Cause.prettyErrors(exit.cause);
if (Cause.hasInterrupts(exit.cause)) {
const interrupted = options.interruptMessage ?? "Operation was aborted.";
const detail = Cause.hasInterruptsOnly(exit.cause)
? ""
: ` ${first?.message ?? Cause.pretty(exit.cause)}`;
throw new SubagentToolInterruptedError(`${interrupted}${detail}`);
}
throw new Error(first?.message ?? Cause.pretty(exit.cause));
}
11 changes: 6 additions & 5 deletions extensions/workflows/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1977,18 +1977,19 @@ export default function workflows(
worktree = created.worktree;
if (!runSettled) record.worktreeBranch = worktree.branch;
}
if (runSignal.aborted || runSettled) {
throw runSignal.reason instanceof Error
? runSignal.reason
: new Error("Workflow was aborted");
}
const agentCwd = worktree?.path ?? requestedCwd;

// Inside the try, not before it: building resources can throw
// (bad settings, an unreadable skills dir), and a throw out here
// would skip the finally and leak the worktree permanently —
// nothing sweeps `.git/pi-worktrees/` afterwards.
try {
if (runSignal.aborted || runSettled) {
throw runSignal.reason instanceof Error
? runSignal.reason
: new Error("Workflow was aborted");
}

let rejectResourceLoad: (() => void) | undefined;
const resourceAbort = new Promise<never>((_resolve, reject) => {
rejectResourceLoad = () =>
Expand Down
8 changes: 7 additions & 1 deletion extensions/workflows/progress-projection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -250,8 +250,11 @@ export class AgentProgressProjection {
snapshot(
toolTimings: ReadonlyMap<string, ProgressToolTiming> = new Map(),
): AgentProgressProjectionSnapshot {
// Reserve the initial task, then spend the remaining byte budget on the
// newest evidence. Forward selection would silently discard final errors
// after enough large tool results, even below the entry-count limit.
const selected = this.firstEntry
? [this.firstEntry, ...this.tailEntries]
? [this.firstEntry, ...this.tailEntries.slice().reverse()]
: [];
const transcript: TranscriptEntry[] = [];
let totalBytes = 0;
Expand All @@ -277,6 +280,9 @@ export class AgentProgressProjection {
: { timestamp: entry.timestamp }),
});
}
// Budgeting order is not display order: keep the retained tail chronological.
const newestFirstTail = transcript.splice(1);
transcript.push(...newestFirstTail.reverse());
if (transcript.length < this.totalEntries) {
transcript.push({
role: "toolResult",
Expand Down
Loading
Loading