fix(storage): bound storage helper probes by the carrier's remaining budget

docs/cli.md

@@ -348,7 +348,9 @@ carriers follow only `HACK_NATIVE_STORAGE_TIMEOUT_MS`. Storage
 proofs that run while a native `up` or `restart` builds and starts workloads do not
 consume the start budget: their elapsed time extends the start deadline, and each
 proof stays bounded by its own carrier budget. Each Docker command inside a carrier
-keeps its fixed 10–15 second cap.
+is bounded by that carrier's remaining budget (reads at most 60 seconds), so a slow
+helper start on a loaded host can use the budget rather than a fixed 15-second cap.
+Helper cleanup keeps its own 10–60 second bound.
 
 The two variables bound different things:
 

docs/reference/native-compose-commands.md

@@ -287,6 +287,21 @@ Hack does not kill a guessed guest PID or automatically stop the application on
 client cancellation. Explicit owned `down` stops the containing instance while
 retaining persistent data.
 
+When the instance has enrolled persistent storage, saved `exec` proves that
+storage before and after the guest command. The opening proof may reuse the
+proof that command storage preparation just made under the same lease; the
+closing proof after the command always runs fresh. Each proof has its own
+`HACK_NATIVE_STORAGE_TIMEOUT_MS` carrier budget. The closing proof is
+fail-closed: if it refuses (including a storage probe timeout, reported as
+`E_COMPOSE_FAILED` with a `storage-*` stage), `exec` fails with that refusal even
+though the guest command already ran and printed its output, and the command's
+exit code is not reported. A storage result that cannot be proven after the
+command must not look like a clean exec. The refusal keeps ownership and saved
+state unchanged, and the next admission proves storage again. Retrying `exec`
+runs the guest command again, so check its effects first when it is not
+idempotent. A cancelled `exec` keeps its known exit code and starts no closing
+proof.
+
 `ps.pending` describes journaled engine mutations, not guest exec sessions. A
 cancelled `exec` can therefore have no pending engine operation even when its guest
 completion is unknown. An interrupted `run` retains its exact one-off and pending

docs/reference/native-compose-storage-xattr-carrier.md

@@ -415,11 +415,27 @@ Budgets are layered and all finite:
   the start deadline but capped in total at 16 × the storage budget or 1 hour,
   whichever is smaller. A guard proof that would start after the cap is spent is
   refused before it runs, and one that ends past it refuses as `storage-budget`.
-- Inside a carrier every Docker probe and command keeps a fixed cap, clipped to
-  the carrier's remaining budget: 15 seconds for reads, target observation and
-  helper create/start, and 10 seconds for helper removal and cleanup reads, which
-  are not clipped. A recorded command's 15-second settlement budget follows its
-  exit.
+- Inside a carrier every Docker probe and command is bounded by the carrier's
+  remaining budget (`nativeComposeStorageProbeTimeoutMs`), not by a fixed cap:
+  - Volume create and helper create/start get the whole remaining budget, so a
+    slow but healthy helper start on a loaded host completes while budget
+    remains, and a hung one times out exactly at the carrier's end. This holds
+    for recorded original commands too (the readonly `verify` proofs that
+    `exec` and `up` run): the child, its publication handshake and its identity
+    reads share that one bound.
+  - Reads and target observation get the remaining budget, at most 60 seconds
+    (the ownership probe's own ceiling).
+  - Helper removal and cleanup reads run even after the carrier's deadline, so
+    they are never clipped to it. Each has its own bound: the remaining budget
+    clamped to 10–60 seconds. A hung engine therefore cannot stall cleanup.
+
+  Earlier fixed caps (15 seconds for reads, target observation and helper
+  create/start; 10 seconds for cleanup) refused healthy `docker start -ai`
+  helpers on loaded hosted runners as `storage-helper-start`/`probe-timeout`
+  while most of the 90-second budget remained (HACK-1369). A timeout that ends
+  before the carrier's deadline still reports `probe-timeout`; one that reaches
+  it reports `storage-budget`. A recorded command's 15-second settlement budget
+  follows its exit.
 - An unrecorded command (an identity read or a legacy root/seed command) waits
   at most three seconds for its process group to become absent after exit. Like
   the handshake and settlement budgets, this deadline is monotonic

src/lib/compose-startup-budget.ts

@@ -79,9 +79,11 @@ export function resolveComposeImageTimeoutMs(
  * Budget of one native storage carrier (one fresh helper proof session), from
  * `HACK_NATIVE_STORAGE_TIMEOUT_MS`. It is separate from the start budget:
  * storage preparation, enrollment and every storage proof use it, never
- * `HACK_COMPOSE_STARTUP_TIMEOUT_MS`. Each Docker probe inside a carrier keeps its
- * own fixed 10-15 s cap, and an exhausted carrier refuses with the closed
- * `storage-budget` effect reason. It grants nothing and relaxes no check.
+ * `HACK_COMPOSE_STARTUP_TIMEOUT_MS`. Each Docker probe inside a carrier is
+ * bounded by the carrier's remaining budget (cleanup keeps its own 10-60 s
+ * bound; see `nativeComposeStorageProbeTimeoutMs`), and an exhausted carrier
+ * refuses with the closed `storage-budget` effect reason. It grants nothing and
+ * relaxes no check.
  */
 export function resolveNativeStorageTimeoutMs(
   opts: {

src/lib/native-compose-command-storage.ts

@@ -357,9 +357,11 @@ export type NativeComposeCommandStorage = Awaited<
 /** Saved exec does not publish a ready receipt. The opening check directly
  * follows command storage preparation under the same lease, with only read-only
  * source and ownership checks between, so it may reuse that adjacent proof. The
- * closing check after the process always performs a fresh proof. Cancellation
- * preserves its known exit and starts no late helper; every future admission
- * re-verifies content. */
+ * closing check after the process always performs a fresh proof and is
+ * fail-closed: its refusal fails exec although the command already ran, and the
+ * command's exit code is not returned (docs/reference/native-compose-commands.md).
+ * Cancellation preserves its known exit and starts no late helper; every future
+ * admission re-verifies content. */
 export async function runNativeComposeStorageVerifiedExec(opts: {
   readonly signal: AbortSignal;
   readonly assertSaved: (

src/lib/native-compose-storage-witness-docker.ts

@@ -116,6 +116,50 @@ const CAPTURE_LIMIT = 4096;
  * uncertainty (HACK-1281). It stays finite and grants nothing.
  */
 const SETTLEMENT_BUDGET_MS = 15_000;
+/**
+ * Floor of every cleanup step of one carrier: cleanup reads after the helper
+ * exits and helper removal. Cleanup runs even after the carrier's own deadline
+ * has passed (it is never clipped to it), so this floor keeps a just-expired
+ * carrier able to remove its helper.
+ */
+export const NATIVE_STORAGE_CLEANUP_FLOOR_MS = 10_000;
+/**
+ * Ceiling of one Docker read probe (`createNativeComposeProbe` accepts at most
+ * 60 s) and of every cleanup step, so a hung engine can stall neither a read
+ * nor cleanup beyond it.
+ */
+export const NATIVE_STORAGE_PROBE_CEILING_MS = 60_000;
+/**
+ * Timeout of one Docker probe or command inside a storage carrier, from the
+ * carrier's remaining `HACK_NATIVE_STORAGE_TIMEOUT_MS` budget (HACK-1369).
+ * Earlier fixed 15 s/10 s caps refused a healthy but slow helper start on
+ * loaded hosts as `probe-timeout` while most of the 90 s budget remained.
+ * - `command` (volume create, helper create/start): the remaining budget, so a
+ *   timeout still fires exactly at the carrier's end.
+ * - `read` (reads and target observation): the remaining budget, at most
+ *   {@link NATIVE_STORAGE_PROBE_CEILING_MS}.
+ * - `cleanup` (post-exit reads and helper removal): its own bound, the remaining
+ *   budget clamped to [{@link NATIVE_STORAGE_CLEANUP_FLOOR_MS},
+ *   {@link NATIVE_STORAGE_PROBE_CEILING_MS}], never clipped to the deadline.
+ * Every value is finite. A non-cleanup value at or below zero is refused by the
+ * probe or command before it spawns; it grants nothing and relaxes no check.
+ */
+export function nativeComposeStorageProbeTimeoutMs(opts: {
+  readonly kind: "read" | "command" | "cleanup";
+  readonly deadline: number;
+  readonly now?: number;
+}): number {
+  const remaining = opts.deadline - (opts.now ?? Date.now());
+  if (opts.kind === "cleanup") {
+    return Math.min(
+      NATIVE_STORAGE_PROBE_CEILING_MS,
+      Math.max(NATIVE_STORAGE_CLEANUP_FLOOR_MS, remaining)
+    );
+  }
+  return opts.kind === "read"
+    ? Math.min(NATIVE_STORAGE_PROBE_CEILING_MS, remaining)
+    : remaining;
+}
 // A late/unconfirmed publication cannot release its original held directory FDs.
 const unsettledCommandFiles = new Set<readonly HeldDirectory[]>();
 type Selection = { readonly name: string; readonly storage: string };
@@ -549,7 +593,10 @@ async function savedCarrierRecovery(opts: {
       await guard();
       const result = await createNativeComposeProbe({
         signal: context.signal,
-        timeoutMs: Math.min(15_000, context.deadline - Date.now()),
+        timeoutMs: nativeComposeStorageProbeTimeoutMs({
+          kind: "read",
+          deadline: context.deadline,
+        }),
       })(args);
       await guard();
       return result;
@@ -1410,9 +1457,10 @@ export async function createNativeComposeDockerStorageXattrCarrier(
       lifetime(context);
     }
     const signal = cleanup ? undefined : context.signal;
-    const timeoutMs = cleanup
-      ? 10_000
-      : Math.min(15_000, context.deadline - Date.now());
+    const timeoutMs = nativeComposeStorageProbeTimeoutMs({
+      kind: cleanup ? "cleanup" : "read",
+      deadline: context.deadline,
+    });
     return nativeComposeStorageIdentityProbe({
       probe: createNativeComposeProbe({ signal, timeoutMs }),
       signal,
@@ -1635,7 +1683,10 @@ export async function createNativeComposeDockerStorageXattrCarrier(
             `io.hack.native-config.storage-provision=${nonce}`,
             selection.name,
           ],
-          timeoutMs: 15_000,
+          timeoutMs: nativeComposeStorageProbeTimeoutMs({
+            kind: "command",
+            deadline: context.deadline,
+          }),
           beforeSpawn: () => {
             tool();
             lifetime(context);
@@ -1774,7 +1825,10 @@ export async function createNativeComposeDockerStorageXattrCarrier(
               program: files.program,
             }),
           ],
-          timeoutMs: 15_000,
+          timeoutMs: nativeComposeStorageProbeTimeoutMs({
+            kind: "command",
+            deadline: context.deadline,
+          }),
           beforeSpawn: () => {
             tool();
             lifetime(context);
@@ -1898,7 +1952,10 @@ export async function createNativeComposeDockerStorageXattrCarrier(
               }
             : undefined,
           args: [docker, "start", "-ai", id],
-          timeoutMs: 15_000,
+          timeoutMs: nativeComposeStorageProbeTimeoutMs({
+            kind: "command",
+            deadline: context.deadline,
+          }),
           beforeSpawn: () => {
             tool();
             lifetime(context);
@@ -1998,7 +2055,10 @@ export async function createNativeComposeDockerStorageXattrCarrier(
             : undefined,
           cleanup: true,
           args: [docker, "rm", id],
-          timeoutMs: 10_000,
+          timeoutMs: nativeComposeStorageProbeTimeoutMs({
+            kind: "cleanup",
+            deadline: context.deadline,
+          }),
           beforeSpawn: tool,
           assertAdmitted: async () => {
             const finalCarrier = await inspectCarrier(true);