EItanya · Oct 14, 2026 · Oct 13, 2026 · Oct 13, 2026 · Oct 13, 2026 · Oct 13, 2026
diff --git a/.ko.yaml b/.ko.yaml
@@ -19,7 +19,7 @@ defaultPlatforms:
   - linux/arm64
 
 baseImageOverrides:
-  github.com/agent-substrate/substrate/demos/sandbox: alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b
+  github.com/agent-substrate/substrate/demos/sandbox: alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
   # atelet needs glibc to run the node's kubelet credential provider, which
   # some platforms ship dynamically linked (e.g. AKS, GKE Ubuntu).
   github.com/agent-substrate/substrate/cmd/atelet: gcr.io/distroless/base-debian13:latest@sha256:389cad21f73e4c37b94ffe5b13736d5a92bd5bd3c6c6b38c2be1c881e14ba2bd
diff --git a/README.md b/README.md
@@ -105,6 +105,7 @@ goals in the near term.
 To quickly set up the complete environment:
 
 1. Make sure you have [Go](https://go.dev/doc/install), [`kubectl`](https://kubernetes.io/docs/tasks/tools/), and [`docker`](https://www.docker.com/) installed and configured on your dev machine. We will automatically manage other dependencies via Go, including [`kind`](https://kind.sigs.k8s.io/).
+   A Linux host that will run more than about a thousand sandboxes also needs a larger kernel neighbor table (`net.ipv4.neigh.default.gc_thresh1/2/3` = 4096/16384/32768); `hack/create-kind-cluster.sh` warns when the host is at the default of 1024.
 
 2. Run the following steps:
 ```shell
@@ -141,15 +142,6 @@ curl -X POST \
    -i http://localhost:8000/
 ```
 
-Worker capacity is versioned: the dataplane (the atelet DaemonSet and the
-worker pods) schedules only on nodes that carry the
-`ate.dev/substrate-version` label, and the install stamps it on every node
-that exists when it runs. A node added later hosts no workers until you label
-it with the installed version
-(`kubectl label node <node> ate.dev/substrate-version=<build version>`).
-`kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version`
-prints the installed version, off the atelet DaemonSet the install created.
-
 ### GKE Quickstart (Development)
 
 1. Create and configure your environment file:
@@ -193,11 +185,6 @@ prints the installed version, off the atelet DaemonSet the install created.
    grant each atespace the namespaces it may read; see
    [Enable it](docs/egress-credential-injection.md#enable-it).
 
-   Nodes that GKE adds later (autoscaling, auto-repair, node upgrades) are
-   born with the node pool's labels, so the pool needs
-   `ate.dev/substrate-version` too; see
-   [Node version labels](tools/setup-gcp/README.md).
-
 5. You can then deploy the sample applications. See [demos/counter/README.md](demos/counter/README.md) or [demos/sandbox/README.md](demos/sandbox/README.md) for detailed walkthroughs.
    ```bash
    ./hack/install-ate.sh --deploy-demo-counter

diff --git a/benchmarking/README.md b/benchmarking/README.md
@@ -53,10 +53,23 @@ convenience:
 ./hack/install-ate.sh --delete-benchmarks
 ```
 
-The installer accepts `--benchmark-worker-count N` (default `1`).
 `--skip-build` and `--worker-memory` are only available when invoking
 `benchmarking/deploy_locust.sh` directly.
 
+### Settings
+
+`ate-setup` reads these settings in addition to those in
+[docs/operator-install.md](../docs/operator-install.md#settings), through the same
+flag, environment variable, and configuration document channels.
+`hack/install-ate.sh` forwards them as `--benchmark-worker-count`,
+`--benchmark-sandbox-class`, and `--benchmark-actor-memory`.
+
+| Key                      | Environment variable          | Flag                       | Default  | Applies to                               |
+|--------------------------|-------------------------------|----------------------------|----------|------------------------------------------|
+| `benchmark.actorMemory`  | `BENCHMARK_ACTOR_MEMORY`      | `--benchmark-actor-memory` | —        | (global)                                 |
+| `benchmark.sandboxClass` | `ATE_BENCHMARK_SANDBOX_CLASS` | `--sandbox-class`          | `gvisor` | `deploy benchmarks`, `delete benchmarks` |
+| `benchmark.workerCount`  | `ATE_BENCHMARK_WORKER_COUNT`  | `--worker-count`           | `1`      | `deploy benchmarks`, `delete benchmarks` |
+
 ## Running Tests
 
 ### Locust Web UI
@@ -88,7 +101,7 @@ not a local entry point. See [automation/README.md](automation/README.md).
 python3 runner.py -f tests/<user-class>.py -t 1m -u 1 --name <run-name> --dest /tmp/bench
 ```
 
-Three flags control the optional post-run measurements described in
+Four flags control the optional post-run measurements described in
 [Benchmark output files](#benchmark-output-files):
 
 * `--cluster-facts` / `--no-cluster-facts`: read node capacity and worker pod
@@ -99,6 +112,11 @@ Three flags control the optional post-run measurements described in
   [Optional: Prometheus + Grafana](#optional-prometheus--grafana).
 * `--atelet-lag-s`: how long to wait after the run before reading the
   atelet's snapshot metrics. Defaults to 70.
+* `--phase-breakdown` / `--no-phase-breakdown`: read the atelet and worker pod
+  logs once the run ends and add the suspend/resume phase percentiles to
+  `stats.jsonl`. On by default; needs `pods/log` access in `ate-system` and
+  `benchmark-workloads` (granted in `automation/manifests/runner-job.yaml.tmpl`).
+  See [analysis/README.md](analysis/README.md#automated-runs).
 
 Test-specific flags are appended to the same command; see the sections below.
 
@@ -305,6 +323,14 @@ wake, the script also shows what an always-on agent costs a system that
 suspends it between events: an internal timer cannot fire in a suspended
 sandbox, so each of those turns needs an external wake.
 
+`personal-assistant-light` is the same day with the RAM arrays scaled
+down to 28Mi and everything else unchanged (files, reads, CPU, dwell and
+the 61 wakes), for runs that want the always-on wake pattern with a small
+memory image: about 40 to 95 MB per pause under gVisor, and under micro-VM
+about 100 MB of guest before the script writes anything, bounded by the
+actor's memory limit. It fits a 256Mi actor. Run it with the same
+fractional think scale.
+
 To run a script of your own without rebuilding anything, hand it to the
 locust deploy:
 
@@ -364,6 +390,124 @@ The web UI shows the same fields; `0` keeps the value boomer-worker started with
 The `actors_per_*` ratios in `trial_summary` are wrong for this test: they
 count users × `--actors-per-user`, not `--total-actors`.
 
+### ResumePaused Benchmark
+
+The ResumePaused benchmark drives ate-api-server's cold resume path at a set
+rate. A cold resume here is a ResumeActor of a paused actor: ateapi takes the
+actor lease, binds a worker, and restores the actor from its snapshot. A
+ResumeActor of a running actor does none of that. Each user owns one actor
+from the `glutton` template. On its first iteration the user creates the actor
+and makes it cold with a resume and a pause; every later iteration resumes the
+actor and pauses it again, so every measured resume is cold.
+
+The warm-up runs in the step that added the user, unpaced, and is recorded
+under its own names.
+
+It sends no requests to the actors themselves, so it runs on the fake data
+plane, where it measures what a cold resume costs ate-api-server and Postgres
+without the router or the restore in the way. The pauses run at the resume
+rate, so a capacity it finds is one of resume and pause cycles, not of resumes
+alone.
+
+Actors are named `rp-<uuid>` and deleted when boomer exits. If the atespace
+cannot be ensured, boomer exits at startup. An actor that fails to warm is
+deleted, and its user creates another on its next iteration. When a step lowers
+the user count, the actors of the stopped users stay paused, unused, until
+boomer exits.
+
+#### ResumePaused Configuration Knobs
+
+* `--target-rps`: Cold resumes per second the whole boomer process paces to
+  (default `0`, unpaced). A `--ladder` step ending in `@target_rps`, as in
+  `200:3m@1000`, changes it as the step starts. Keep the users at or above the
+  rate times the p99 of one resume plus one pause in seconds, or the step falls
+  short of its rate.
+
+The web UI shows the same field.
+
+Trim the start of each ladder step. A rate change reaches boomer at its next
+poll of the master, within 10 s, and a step that adds users carries their
+warm-up until they have all joined at the spawn rate.
+
+#### ResumePaused Reported Metrics
+
+* `ResumeActorCold`: Latency of each measured resume. A resume that finds its
+  actor already running fails, as no cold resume happened.
+* `PauseActor`: Latency of the pause that makes the actor cold again.
+* `PauseActorRecovery`: After a failed cycle, the pause that makes the actor
+  cold for the next one. It is load on the server, so it is recorded, but
+  apart from `PauseActor`. A recovery pause that finds the actor already
+  paused counts as a success.
+* `CreateActor`, `ResumeActorWarmup`, `PauseActorWarmup`: A user's warm-up of
+  its actor, in the step that added the user.
+
+### Isolate Mode
+
+A real cold resume is limited by the node's restore capacity, which hides the
+limits of ate-api-server and Postgres. Isolate mode takes the nodes out by
+installing substrate with a fake data plane:
+
+```bash
+hack/install-ate.sh --deploy-ate-system --fake-data-plane
+./benchmarking/workloads/deploy.sh --deploy --worker-count 200 \
+  --fake-atelet-env FAKE_ATELET_DELAY=500ms
+```
+
+`deploy.sh` deploys the usual WorkerPool, but the install serves it with two
+fakes from `cmd/benchmarking/isolate/` instead of worker pods:
+
+* `fake-workersync` stands in for ate-controller's WorkerPool controller and
+  worker syncer. It honors each pool's `spec.replicas` with fake Workers, none
+  backed by a pod, round robin on the nodes that run fake-atelet, and scales them
+  up and down with the pool, draining a Worker before deleting it. Each Worker
+  reports what a real worker would: cpu and memory from the pool's
+  `template.resources.limits`, or the node's allocatable without one, and 1000
+  actors, ateom's default, unless the pool's `ate.dev/fake-max-actors`
+  annotation says otherwise. It writes the pool's status, so
+  `kubectl get workerpool` and `kubectl scale workerpool` work. A node keeps
+  its Workers for `--node-grace`, five minutes by default, after its
+  fake-atelet pod goes, so a DaemonSet rollout does not move them; past that
+  they move to a node that still runs fake-atelet.
+* `fake-atelet` takes atelet's place in its DaemonSet, on every node atelet
+  would run on, and answers every
+  lifecycle call with success after its `--delay`, without running or
+  saving any workload. It writes one placeholder object per snapshot, because
+  ate-api-server refuses to tag an empty one. Like ateom, it mints the actor's
+  certificate from ate-api-server on every run and restore that names an
+  egress gateway, which the default install does; the mint is ate-api-server's
+  time, so it adds to `--delay`, the data plane's share of the call.
+  It also relays fake-workersync's capacity reports, as atelet relays ateom's:
+  ate-api-server accepts a report only from the atelet on the Worker's node.
+  The relay admits only fake-workersync's pod identity.
+
+The install also scales ate-controller to zero, whose worker syncer would
+delete the fake Workers. Its other controllers stop with it: no NetworkPolicy
+is made for a WorkerPool, and the egress MITM trust bundle is not derived.
+Fake actors need neither. fake-atelet takes the storage settings of atelet's
+objectstore-plugin sidecar, whose writes it takes over, so its snapshot
+placeholders go to the storage the install points atelet at. The install
+refuses a cluster whose WorkerPools still have worker pods, which would
+outlive ate-controller. A normal `hack/install-ate.sh --deploy-ate-system`
+puts the real data plane back, and `--delete-ate-system` removes the fakes
+either way. Never use the mode on a
+cluster that serves real actors: actors placed on fake Workers do not exist.
+
+The installer only chooses the fake data plane; the benchmark tunes it.
+fake-atelet reads each tuning flag from a `FAKE_ATELET_<FLAG>` variable in the
+optional `fake-atelet-tuning` ConfigMap, which no install creates or resets:
+`FAKE_ATELET_DELAY` sets `--delay`, `FAKE_ATELET_DELAY_RESTORE` overrides it
+for Restore alone, and `FAKE_ATELET_MINT_ACTOR_CERTIFICATE=false` skips the
+certificate mint. `deploy.sh --fake-atelet-env NAME=VALUE`, repeatable, writes
+that ConfigMap to exactly the given entries and restarts fake-atelet when it
+changes, before the WorkerPool is applied.
+
+Nothing answers behind the router, so the mode suits only a user class that
+sends the actors no requests, such as `ate_api.py`, which creates, gets,
+resumes and suspends actors. Every boomer user class pings its actors through
+the router. In `tests.yaml`, an `isolate` block selects the mode, and its
+keys become `deploy.sh --fake-atelet-env` entries: `delay` sets
+`FAKE_ATELET_DELAY`, and so on.
+
 ### Viewing Traces
 You must have enabled otel tracing for your cluster to view traces.
 
@@ -382,7 +526,10 @@ them are checked into the repository.
 * `logs.txt`, `traces.txt`: the runner log, and the trace IDs seen during the run.
 * `stats.jsonl`: one JSON object per line, one per metric. Every row carries
   the same five keys: `timestamp`, `tag`, `test_name`, `metric`, and a flat
-  `measurements` map holding that metric's numbers.
+  `measurements` map holding that metric's numbers. The `phase_*` rows and
+  the `phase_breakdown_summary` row are the suspend/resume phase percentiles
+  read from the node logs (see
+  [analysis/README.md](analysis/README.md#automated-runs)).
 * `server_summary.json`: server-side telemetry harvested from Prometheus,
   including the per-sample bin-packing timeseries.
 

diff --git a/benchmarking/analysis/README.md b/benchmarking/analysis/README.md
@@ -12,8 +12,8 @@ Two joinable JSON log records feed it, each written by both layers:
 
 | record (`msg`) | emitter | keys |
 |---|---|---|
-| `Restore timing breakdown` | atelet and ateom-microvm | `ate.actor.restore.duration.<phase>` / `ateom.actor.restore.duration.<phase>` |
-| `Checkpoint timing breakdown` | atelet and ateom-microvm | `ate.actor.checkpoint.duration.<phase>` / `ateom.actor.checkpoint.duration.<phase>` |
+| `Restore timing breakdown` | atelet, ateom-microvm and ateom-gvisor | `ate.actor.restore.duration.<phase>` / `ateom.actor.restore.duration.<phase>` |
+| `Checkpoint timing breakdown` | atelet, ateom-microvm and ateom-gvisor | `ate.actor.checkpoint.duration.<phase>` / `ateom.actor.checkpoint.duration.<phase>` |
 
 Every record carries the full actor identity (`ate.actor.uid`, name,
 atespace, template) and the snapshot scope, which the histograms are barred
@@ -74,13 +74,18 @@ A failed atelet operation still writes its record, marked with `error.type`
 **Phase percentiles.** Per layer, operation, sandbox class, snapshot kind,
 scope and phase: count, p50, p90, p95 and max. A `golden` restore downloads
 the golden image and a `latest` one the actor's own, so they are separate
-rows, as are gVisor and micro-VM checkpoints. The atelet rows split a checkpoint between
-`sandbox_assets`, `ateom_checkpoint` and `persist`, and a restore between
-`volume_mount`, `manifest_fetch`, `sandbox_assets`, `download`, `oci_unpack`
-and `ateom_restore`. The ateom rows split the `ateom_*` phase further:
+rows, as are gVisor and micro-VM operations. The atelet rows split a
+checkpoint between `sandbox_assets`, `ateom_checkpoint` and `persist`, and a
+restore between `volume_mount`, `manifest_fetch`, `sandbox_assets`,
+`download`, `oci_unpack` and `ateom_restore`. The ateom rows split the
+`ateom_*` phase further with the runtime's own phases: for micro-VM
 `prep` / `pause` / `snapshot` / `durable_dir` / `rootfs_upper` / `teardown`
-for a checkpoint, `prep` / `bundles` / `upper_join` / `lowers` / `tap` /
-`vmm_launch` / `vm_restore` / `resume` / `wakeup_probe` for a restore.
+on a checkpoint and `prep` / … / `vm_restore` / `wakeup_probe` on a restore;
+for gVisor `pause` / `checkpoint` / `durable_dir` / `resume` / `teardown` and
+`prep` / `net_setup` / … / `pause_restore` / `app_restore` / `wakeup_probe` /
+`activate` (the full lists are in `cmd/ateom-*/phaselog.go`). The ateom
+records carry no sandbox class or kind of their own; a paired one takes both
+from its atelet record, so both layers split into the same rows.
 
 Concurrency matters when reading them: the atelet restore phases overlap
 (the download runs alongside the asset fetch and OCI unpack), the three
@@ -107,6 +112,27 @@ every time are systematic; different phases each time are environmental.
 The parser is deliberately tolerant: it scans any line for a JSON object and
 matches on `msg`, so raw `kubectl logs` dumps (even with prefixes) work.
 
+## Automated runs
+
+The locust runner (`benchmarking/locust/runner.py`) does the collection
+itself at the end of a headless run: `phase_breakdown.py` reads the atelet
+and worker pod logs through the Kubernetes API while the pods still exist
+(the orchestrator deletes them as soon as the runner exits) and appends the
+percentiles to the run's `stats.jsonl`, one row per layer / operation /
+class / kind / scope / phase (`metric: phase_<layer>_<op>_<class>_<kind>_<scope>_<phase>`;
+the measurements map carries those dimensions as fields next to `count`,
+`p50_ms`, `p90_ms`, `p95_ms`, `max_ms`, all as strings like the runner's
+other rows). A `phase_breakdown_summary` row is always written: pods read
+and failed, the record count, and the atelet record counts next to locust's
+`SuspendActor` / `ResumeActor` request and failure counts. More records than
+requests is normal (boomer suspends its actors on shutdown); fewer records
+than successful requests means records were lost, to pods that could not be
+read or to a node log rotated during the run, and the runner's log says
+which. `--no-phase-breakdown`
+skips it. The runner's service account needs `list` on `pods` and `get` on
+`pods/log` in `ate-system` and `benchmark-workloads`
+(`automation/manifests/runner-job.yaml.tmpl`).
+
 ## Tests
 
 ```bash