Repository navigation
Sync upstream snapshot lifecycle and control-plane updates - #77
Merged
Merged
Conversation
…ER (agent-substrate#1976) The `:8080` listener follows `OTEL_METRICS_EXPORTER`, per the policy in agent-substrate#2067. Unset keeps push and `:8080`, `otlp` closes `:8080`, `prometheus` serves only, `none` does neither. Fixes agent-substrate#933 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
The protobuf style guide treats an abbreviation as a single word in
message, enum, service and RPC names (HttpRule, not HTTPRule).
Our protos mixed that with Go-style all-caps initialisms, so generated
Go code mixed TitleCase field names with all-caps type names. For
example:
```
Https: &ateapipb.HTTPSRule{...}
```
Handwritten Go keeps Go's initialism rule (HTTP, TLS, JWT). Only names
generated from protos use TitleCase. It's not ideal, but it's likely the
least bad option.
Fixes agent-substrate#2405
…y /metrics (agent-substrate#2424) ateapi, atelet, atenet-router, atenet-egress, and the kubernetes-secrets credential provider served `/readyz` and `/healthz` on the Prometheus `/metrics` server on `:9090`. A later change lets `OTEL_METRICS_EXPORTER` turn off `/metrics`, and that change must not stop the kubelet probes. This PR moves the probes to a separate health server. ## Breaking change The health probes move from `:9090` to `:9091` (`/readyz`, `/healthz`). `:9090` now serves only `/metrics`. If you use your own manifests, change your probes to the new `health` port. Apply the new images and manifests together. atelet no longer has `hostPort: 9090`. ## Testing - `go test -race` passes for `internal/serverboot`, `cmd/ate-setup/internal/steps`, `cmd/ateapi`, `cmd/atelet`, `cmd/atenet/internal/router`, and `cmd/credential-provider/kubernetes-secrets`. - All the changed binaries build. Part of agent-substrate#2067 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…lete reports the delete operation (agent-substrate#2356) ## Behavior changes - `ate.actor.crashes`: when ateapi loses the worker of an actor in DELETING (possible with `any_state`), the crash now has `ate.actor.operation.name=delete`. Before this change it had `unknown`. - The crash message in the actor status now starts with `delete failed: ` in that case. Before this change it had no prefix. - `ate.actor.lifecycle.operation.duration`: when resume, suspend, pause, or revert fails before ateapi loads the actor, the series now omits `ate.template.name` and `ate.template.atespace`. Before this change it had them as the empty string. - `rpc.server.call.duration` and `rpc.client.call.duration` now list `rpc.system.name` in the registry, and the client histogram also lists `server.address` and `server.port`. This is only a change to the docs. otelgrpc already sent these keys.
…ent-substrate#1997) Fixes part of agent-substrate#1709 This adds declarative validation to atelet's AteomHerder RPCs and to the WorkloadSpec they carry. Malformed requests are now rejected with `InvalidArgument` before atelet builds any host path or touches an ateom. The hand-written validators they replace are removed. With this, every atelet RPC request is validated, so the generator now selects them by the `Request` suffix, as controlapi does. | RPC | Rules | |---|---| | Run | target ateom UID, atespace and actor name are required short names; actor UID is a required UUID; template identity, when set, is short names; spec is required and validated in full; `cpu_milli` is 0..999999 and `memory_bytes` ≥ 0; egress gateway address, when set, is a host:port; sandbox assets are required | | Terminate | same identity rules; spec is optional (delete of template-less actors) and validated in full when set | | Checkpoint | same identity rules; spec required; type is LOCAL or EXTERNAL and the matching config must be set; local snapshot name is a short name; external snapshot URI is parseable; scope is FULL or DATA | | Restore | same as Checkpoint, plus: scope may also be DATA_ON_GOLDEN, and `base_config` is required exactly then; same size, egress and sandbox asset rules as Run | | UploadPausedCheckpoint | same identity rules, but the golden atespace is rejected; local snapshot name is a short name; destination URI is parseable; desired scope is FULL or DATA | **WorkloadSpec** - Containers: names are unique short names, and `pause` is reserved. Images are pinned by digest, the same check ateapi applies when a template is created. Args, env, working dir, probes and mounts are bounded. - Volumes: each volume has exactly one source; external and image sources are checked; system-info file paths are unique; at most one `actor_metadata` data source is allowed (the same rule is added on the ateapi side). **Sandbox assets:** the sandbox class is gvisor or microvm; the pause image is pinned by digest (stricter than the SandboxConfig CRD's `@` check); asset URLs are bounded; hashes are SHA-256. **Wire format:** the `Volume`, `SystemInfoDataSource`, Checkpoint `config` and Restore `config` oneofs become plain fields with the same field numbers, so the wire format doesn't change. They're validated as unions, because validation tags can't be placed on oneof members. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…gent-substrate#2497) `net.ipv4.neigh.default.gc_thresh3` is global across network namespaces; at its default of 1024 a host with about a thousand sandboxes overflows the neighbor table and connections to sandboxes time out. - `hack/create-kind-cluster.sh` warns on Linux hosts at or below 1024 and prints the values to set; it changes nothing. - One sentence in the README. `shellcheck` clean.
…ubstrate#2549) Fixes agent-substrate#2528 Fake plugins were exec'd while a concurrent fork still held their write fd (ETXTBSY). Writes now hold `syscall.ForkLock`, in one helper. Under CPU load with `-race`: 13 failures in 600 runs before, 0 after. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…te#2531) - Adds --upstream-connect (default off). The Envoy router sends actor traffic through mTLS CONNECT tunnels to atunnel's 8443 listener. The :443 (non connect) path is unchanged. - Tunnels are per worker, actor and port, and reused across requests. gRPC goes over HTTP/2, everything else over HTTP/1.1. - atunnel now closes a tunnel when the actor closes its connection, so a reused tunnel can't swallow the next request. - With the flag on, atunnel's 421 and 502 reach clients as 503. - A follow-up PR will make CONNECT the default. Fixes agent-substrate#1309 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…onl from the runner (agent-substrate#2148) ## benchmarking: append the suspend/resume phase percentiles to stats.jsonl from the runner Stacked on agent-substrate#2009 (base branch `phase-logs-analysis` until it merges). ### Why agent-substrate#2009 adds the tool that turns the `Checkpoint/Restore timing breakdown` records into percentiles, but nothing in the automated runs calls it: the orchestrator deletes the worker and ate-system pods as soon as a test's runner exits, so the records never reach `stats.jsonl` or anything that reads it. The runner is the one process that holds the run's identity, owns `stats.jsonl`, and is still alive while the pods are — so it does the collection itself, the way it already collects cluster facts. ### What - **`locust/phase_breakdown.py`** (new): after locust finishes, lists the `app=atelet` pods in `ate-system` and the `ate.dev/worker-pool` pods in `benchmark-workloads` through the Kubernetes client, reads their logs since the run started (a restarted container's previous log too), parses the records with `phase_report`, and appends to `stats.jsonl`: - one row per layer / op / sandbox class / kind / scope / phase — `metric: phase_<layer>_<op>_<class>_<kind>_<scope>_<phase>`, `measurements`: those dimensions as fields plus `count`, `p50_ms`, `p90_ms`, `p95_ms`, `max_ms`, all strings like the runner's other rows; - one `phase_breakdown_summary` row: pods read and failed, record count, and the atelet checkpoint/restore record counts next to locust's `SuspendActor`/`ResumeActor` request counts. Fewer records than requests means a node log rotated during the run; the runner's log says so. Additive like the cluster facts: a failure here never costs the locust rows. - **`locust/runner.py`**: `--phase-breakdown` (default on, `--no-phase-breakdown` to skip), called after the cluster-facts block. - **`automation/manifests/runner-job.yaml.tmpl`**: `benchmark-runner` gains `get`/`list` on `pods` and `get` on `pods/log` in both namespaces. - **`locust/Dockerfile`**: copies `phase_breakdown.py` and `analysis/phase_report.py` into the image. - **`analysis/phase_report.py`**: `parse_lines()` and `stats_rows()` for the above (the CLI is unchanged). - Docs: `analysis/README.md` "Automated runs", the flag in `benchmarking/README.md`, a pointer in the runbook. ### Testing - `python3 -m unittest discover -s benchmarking/locust/unit_tests` — 18 tests (4 new, mocked `CoreV1Api`: rows appended in shape, rotation warning, unreadable pod skipped, restarted container read twice, empty run still writes the summary, failed requests don't trigger the warning). - `python3 -m unittest discover -s benchmarking/analysis` — 22 tests. - Live: against a GKE cluster with 3 workers, after a 10-cycle glutton run, `append_phase_breakdown` read the six pods and wrote 46 phase rows plus the summary (12 atelet checkpoints vs 10 locust suspends + 2 shutdown suspends). The template's RBAC was applied and a log read impersonating `benchmark-runner` succeeded in both namespaces.
…te#2438) ## Summary Align the two remaining nonconforming public Timestamp field names with the API's AIP naming conventions: - `MintActorJwtResponse.expires_at` becomes `expire_time`. - `GoldenSnapshotStatus.take_golden_snapshot_at` becomes `snapshot_time`. Update in-repository callers and existing tests, regenerate protobuf and declarative validation code, and document the `_time` naming rule. Retain the existing validation, lifecycle, and JWT functional coverage; the additional protobuf serialization test file has been removed. Fixes agent-substrate#2415. ## Breaking change Regenerate clients and update references to these fields: | Before | After | | --- | --- | | `expires_at` / `expiresAt` | `expire_time` / `expireTime` | | `take_golden_snapshot_at` / `takeGoldenSnapshotAt` | `snapshot_time` / `snapshotTime` | | Go `ExpiresAt` / `GetExpiresAt()` | `ExpireTime` / `GetExpireTime()` | | Go `TakeGoldenSnapshotAt` / `GetTakeGoldenSnapshotAt()` | `SnapshotTime` / `GetSnapshotTime()` | Timestamp semantics, types, and protobuf field numbers are unchanged. Old-name aliases are not retained, following the project's pre-1.0 compatibility policy. Please apply the `breaking-change` label. ## Verification - [x] Race-enabled tests for the retained validation, lifecycle, and egress JWT packages; the public protobuf package builds. - [x] `TestMintActorJWT` through the functional API harness. - [x] Repository Linux-aware Go lint. - [x] Validation coverage check. - [x] `hack/verify-all.sh`: all 15 current repository verifiers passed on the latest conflict refresh. - [ ] Full Linux/root/infrastructure test matrix (not run locally). ## CI follow-up The October 12 E2E failure was two timeouts to public `httpbin.org` in the credential-injection suite. Current `main` includes agent-substrate#2535, which replaces that external origin with an in-cluster echo server and restores the test after the earlier upstream quarantine. This PR adds no test skip of its own. ## Contribution note This is a first contribution, opened as a draft for coordination and feedback. GitHub Copilot CLI was used to help prepare the code, tests, and PR description.
…ser class (agent-substrate#2350) This PR lets boomer hold a fixed request rate for each ladder step, and adds a user class, `resumepaused`, that drives ate-api-server's cold resume path at that rate. It is needed for ate-api-server capacity runs: a closed-loop user can't hold a rate, because the offered load drops whenever the server slows down. Benchmark-only, with no production code changes. ## Commits 1. **Pace boomer to a target request rate per ladder step.** A new dynconfig field, `target_rps`, is set with `--target-rps` or per ladder step as `users:duration@rps` (e.g. `200:3m@4000`). Boomer picks up a change within 10 s. Only user classes that pace themselves honor it, which today is resumepaused. 2. **Add the resumepaused user class.** Each user owns one actor. On its first iteration the user creates the actor and makes it cold with a resume and a pause; every later iteration resumes it (`ResumeActorCold`, paced to `target_rps`) and pauses it again (`PauseActor`). The warm-up is unpaced and recorded under its own names (`CreateActor`, `ResumeActorWarmup`, `PauseActorWarmup`) in the step that added the user. Every actor is deleted on exit. It sends no traffic to the actors, so it runs on the fake data plane (agent-substrate#2300). ## Notes for reviewers - A capacity resumepaused finds is one of resume and pause cycles, not of resumes alone. - Keep `users` at or above the rate times the p99 cycle time, or a step falls short of its rate. - `target_rps` paces one boomer worker. - A step that adds many users carries their warm-up load while they join at the spawn rate, so trim the start of each step. When a step lowers the user count, the stopped users' actors stay paused until boomer exits. - The `tests.yaml` capacity tests come in a follow-up PR. Details are in the ResumePaused section of `benchmarking/README.md`. ## Testing - `go test -race ./internal/benchmarking/boomer/...` - `python3 benchmarking/locust/unit_tests/test_ladder_shape.py` (needs locust) - `python3 benchmarking/automation/test_orchestrator.py` Addresses agent-substrate#2261 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…bstrate#2561) `TestActorEgressMITMTrust` fetched `https://example.com/` through the gateway's MITM leg and required a 200. When that public origin answered 503 or slowly, the e2e lanes failed on pull requests that did not touch egress. The credential injection suite had the same exposure to `httpbin.org` and already moved to the in-cluster echo server; this does the same for the MITM suite. The intercepted fetches (the trust proof, the system-roots negative, and the Host-port cases) now reach `echo-https` and `echo-http`, testserver origins deployed in the cluster. Their certificate comes from the service-DNS signer, which the gateway trusts after `hack/setup-e2e-egress-tls-kind.sh`; CI already runs that before both lanes. TLS passthrough keeps its public origin, since the proof that the relay did not intercept is the origin's own certificate validating against the system roots. Those fetches now retry transport errors and 5xx for a bounded window and then fail. They never skip, per `docs/dev/best-practices/ci-fail-closed.md`. Verified with `gofmt`, `go vet ./internal/e2e/...`, `go test -c` on the suite, `hack/verify/boilerplate.sh`, and `golangci-lint` on `./internal/e2e/...`. The suite itself needs a cluster and did not run locally; the e2e lanes in CI run it in both sandbox classes.
… label their namespace ate.dev/api-access=true (agent-substrate#2522) Add default-deny ingress NetworkPolicies for the ate-system control plane, with allow rules for the flows it needs: - ate-api-server :443 from ate-controller, atelet, atenet-router, atenet-egress, and namespaces labeled ate.dev/api-access=true - postgres :5432 and atelet gRPC :8085 from ate-api-server only - atenet-egress https from worker pods (ate.dev/worker-pool) in any namespace - atenet-router data ports (http, https, connect, connect-tls) from any source - metrics and health ports from any source (probes, scrapers) - kind: rustfs S3 API from ate-api-server, atelet and the bucket-init Job The router's /statusz port (4040), the rustfs console, and every other port are reachable only through kubectl port-forward. Breaking change: in-cluster clients of the Substrate API outside ate-system must label their namespace ate.dev/api-access=true. GKE Gateway health checks pointed at the router's status port (4040, /statusz) must move to the metrics port (9090, /readyz). The benchmarking namespace is labeled. I want to get this in for M3 because all exposed APIs become customer dependencies unless we restrict them. Fixes agent-substrate#2278 - [X] Tests pass - [X] Appropriate changes to documentation are included in the PR
This picks up agent-substrate#2297 from @aiqubits, rebased on main, with the e2e failure fixed. The router's unauthenticated /statusz dashboard shows recent cross-tenant requests, worker destinations, and process configuration to any pod that can reach it. This binds it to 127.0.0.1 by default through a new --status-address flag and removes its port from the router Service. Admins can still get to it with kubectl -n ate-system port-forward deployment/atenet-router 4040:4040. On top of the original change, the e2e parking suite now port-forwards to the status port on a router pod rather than the Service, since the Service no longer publishes it. #### Breaking change GKE HealthCheckPolicy configs using port 4040 and /statusz should move to port 9091 and /readyz. The router Deployment no longer declares the status container port either. Fixes agent-substrate#2275 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Co-authored-by: aiqubits <aiqubits@hotmail.com>
…gent-substrate#2564) CI (`go test -race`, Linux) fails `cmd/atelet` with `TestSandboxAssetPrewarmDownloads ... testing.go: TempDir RemoveAll cleanup: unlinkat /tmp/TestSandboxAssetPrewarmDownloads.../002/layers/sha256: directory not empty`. The test returns as soon as the runsc asset lands and joins the prewarm worker, but the pause-image pull the worker started is not joined. `imagecache.Store.EnsureImage` runs pulls detached from the caller's context, so a canceled caller does not fail the other waiters, and it returns to the worker on cancellation while the pull keeps unpacking layers into the store. The TempDir cleanup then races that unpack. The test now looks the pause image up by digest after the asset lands. The lookup joins the in-flight pull through the store's singleflight, so no write outlives the test. Under CPU pressure with the race detector on Linux, 1 of 100 and 1 of 500 runs failed before the change and 0 of 1500 after. --------- Co-authored-by: Lior Lieberman <liorlieberman@google.com>
Use pre-existing plugin scripts in testdata/ instead of writing executable files during parallel tests, avoiding ETXTBSY races when concurrent subtests fork and exec plugins (https://go.dev/issue/22315).
…substrate#2568) Fixes agent-substrate#2303 - `envoyproxy/envoy` v1.39.1 → v1.39.4 (libssl3 `…1.29` → `…1.30`), everywhere it was pinned. - `alpine` 3.24.1 → 3.24.2 for `demos/sandbox` (OpenSSL 3.5.7 → 3.5.8). - `hack/tools`: `golang.org/x/*` to latest. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>
Fixes agent-substrate#2409 #### Breaking change Go clients of `pkg/proto/ateapipb`: the `ContainerWakeupProbe` message is renamed to `WakeupProbe`. Replace `ateapipb.ContainerWakeupProbe` with `ateapipb.WakeupProbe`. The field name (`wakeupProbe`) and the wire format are unchanged, so manifests, JSON requests and stored templates need no change.
…d one (agent-substrate#2588) `ate-setup` doesn't remember an external PostgreSQL's connection string between runs. The install record leaves it out intentionally because it can carry a password. If a run forgets `ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING`, `ate-setup` assumes the bundled database and writes its address into the API server's Secret: - `deploy apiserver` fails waiting for a bundled PostgreSQL that doesn't exist, and the next API server restart can't reach the database. - `deploy ate-system` starts an empty bundled PostgreSQL and moves the API server onto it, so every actor looks gone. ------ Now, if the run names no database and the Secret already holds an address that isn't the bundled one, `ate-setup` stops before it changes the database settings and names the missing variable. A fresh installs, bundled installs and Cloud SQL work as before, and so does leaving Cloud SQL with an empty `ATE_API_POSTGRES_CLOUDSQL_INSTANCE`. A second commit does the same for `ATE_API_POSTGRES_OWNER_CONNECTION_STRING`. If the install uses a separate owner login and a run gives only the read-write string, ate-setup stops instead of replacing the owner login. Found while reviewing the upgrade runbook (agent-substrate#1844). Part of agent-substrate#1686 - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…1812) `ate-setup` no longer labels nodes with `ate.dev/substrate-version`. The atelet DaemonSet is now named `atelet` and runs on every node, instead of `atelet-<version suffix>` pinned to nodes carrying its version. The label let two Substrate versions run side by side on disjoint sets of nodes, which the clone-WorkerPool upgrade runbook relied on. The GA upgrade doesn't split nodes by version: atelet rolls in place and worker pools roll as ordinary Deployment updates. With the label, deploying a new atelet creates a second DaemonSet pinned to a version no node carries, so it schedules nowhere. This merges together with agent-substrate#1844, which replaces the label-based runbook in `docs/upgrade.md`. #### Breaking change Due to the removal of version label system. An install from v0.4 or earlier can't be upgraded in place. Reinstall it, as with every pre-GA minor release. Part of agent-substrate#1686 - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR
…#2582) Embed the actor atespace, name, and UID in the same X.509 extension used by actor certificates so both credential types identify the actor incarnation consistently. Signed-off-by: Christian Posta <christian.posta@gmail.com>
Change the kubectl-ate `-v` flag from "version" to "verbose". It can be stacked (-vvv). Then use it in various ways - soft errors (internally retried) and gRPC dumping, for example. #### Breaking change The meaning of `-v` changes from "version" (defined by cobra internals) to "verbose" (by explicit definition). `--version` remains. - [X] Tests pass - [X] Appropriate changes to documentation are included in the PR xref agent-substrate#540
…substrate#2218) `ateom-microvm` attaches the kata guest image (`rootfs.img`) as a read-only `virtio-blk` disk. Every file the guest reads from it (such as `kata-agent` and its libraries, ~35 MiB by the end of a cold boot) is copied into the guest page cache. Because that copy lives in guest RAM, every actor holds its own copy and every memory snapshot stores it. Attach the guest image over `virtio-pmem` with `discard_writes: true` and mount its root filesystem with DAX (`root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4`). Under DAX the guest reads and executes the image's files directly from the host mapping instead of copying them into guest RAM, so those pages sit once in the node's page cache and stay out of every actor snapshot. ## Changes - `cmd/ateom-microvm/internal/ch/createvm.go`: replace `DiskConfig` / `VmConfig.Disks` with `PmemConfig` / `VmConfig.Pmem`. - `cmd/ateom-microvm/run.go`: attach the guest image as a `PmemConfig` with `DiscardWrites: true` and mount `root=/dev/pmem0p1` with `rootflags=dax,data=ordered,errors=remount-ro`. ## Impact (measured) A/B on a `c3-highmem-192-metal` GKE node (amd64), 3 runs per build, replaying a 21-step coding-agent trace (SWE-bench `astropy-7336`) in a microVM (1 vCPU, 1920 MiB guest RAM) across 4 local pause/resume cycles followed by 4 object-storage suspend/resume cycles: | Metric | virtio-blk | virtio-pmem + DAX | Change | |---|---|---|---| | Golden snapshot memory, compressed | 38.3 MB | 25.6 MB | −12.7 MB (−33%) | | Snapshot after last suspend: populated memory | 256 / 256 / 256 MB | 227 / 228 / 227 MB | −29 MB (−11%) | | Snapshot after last suspend: compressed in object storage | 111 / 111 / 112 MB | 99 / 99 / 98 MB | −13 MB (−11%) | | Resume from local pause checkpoint (median) | 516 ms (504–529) | 481 ms (477–484) | −35 ms (−7%) | | Resume from object storage (median) | 1,034 ms (1,014–1,063) | 959 ms (939–977) | −75 ms (−7%) | ## Testing - `gofmt`, `go vet`, and `go test -race ./cmd/ateom-microvm/...` pass. - End-to-end A/B benchmarked on a microVM cluster as shown above. Fixes agent-substrate#2201 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…-substrate#2592) Part of Actor Lifecycle v2 agent-substrate#2069 Restructures the atelet Checkpoint, Resume, and UploadPausedCheckpoint RPCs to organize the snapshot fields into a new Snapshot message, along with a new UploadMode to indicate if the upload should be synchronous or asynchronous. The latest snapshots are also added to the responses for reporting which snapshots have been uploaded (asynchronously). This PR is an API rename, there are no behavior changes. Pause will use the new deferred upload mode (which is unimplemented), and Suspend uses the new immediate mode. #### Breaking change atelet Checkpoint, Resume and UploadPausedCheckpoint APIs use new Snapshot messages. UploadPausedCheckpoint has been renamed to PromoteSnapshot. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ent-substrate#2572) The operator install page listed settings that only configure demos and benchmarks, which is confusing for an operator reading it as installer documentation. Move them to the counter, claude-code-multiplex, and benchmarking READMEs, and have the doc test check each setting against the document that owns it. Fixes agent-substrate#2569
… external dependencies (agent-substrate#2590) agent-substrate#2513 Egress e2e tests fetched example.com/.org/.edu/.net, so an outage or CA change on those sites failed CI for every PR. The networking tests also told MITM from passthrough by matching `O=Google Trust Services` in a third party's certificate. This moves every egress e2e test onto test-server pods it deploys itself: Changes: - **One pod, both schemes.** testserver `http`/`websocket` now always serve cleartext on `--listen` and, given a certificate, also HTTPS on `:8443`. `ServerPod.TLSPort` publishes it (replacing `HealthScheme`). `ServiceDNSTLS` gives the pod a certificate from the service-DNS signer. - **Trust-based proofs.** The gateway trusts the service-DNS CA (via `hack/setup-e2e-egress-tls-kind.sh`); actors don't. A verified fetch with the actor's default trust proves interception. A fetch trusting only the service-DNS CA proves passthrough. - **networking:** `TestActorEgressHTTPS` uses an in-cluster origin. The two hostname tests merge into `TestActorEgressHTTPSByHostname`, whose denial check now fails on an x509 error, so a leak to passthrough can't count as a denial. - **egressmitm:** in-cluster origins; denials use `.cluster.local` aliases, which no exact-match rule lists. A new step checks that the projected bundle carries the system roots (`WithSystemRoots()`). - **Existing TLS origins** move to `TLSPort`; egresscredinject's echo pair becomes one pod. `DeploySubstrateFixture` takes trust-bundle names instead of a bool, which updates its callers mechanically. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…substrate#2216) ## Problem `unpackLayer` (`internal/imagecache/unpack.go`) does not restore the `mtime` recorded in a layer's tar headers. Several inefficiencies: - **Unnecessary recompilation**: cached build artifacts in the image (e.g. `.pyc` files) won't be trusted and become stale because the `mtime` changed. All imports need recompilation (on every actor). - **Larger `rootfs upper` and snapshots**: recompiled files are written into each actor's `rootfs upper`, making `rootfs upper` larger than it should be --> larger snapshots ## Changes - Regular files get the mtime of their tar entry as soon as they are written. - Directories get theirs in the final pass that already restores directory modes, after all of their children are written, so read-only directories are covered too. - Symlinks keep the unpack time, because `Chtimes` follows the link. Hard links share their target's inode. - New test: `TestUnpackLayer_RestoresModTimes`. This matches how `containerd` unpacks OCI image layers in [`pkg/archive/tar.go`](https://github.com/containerd/containerd/blob/v2.0.0/pkg/archive/tar.go#L305-L325). ## Improvement (measured) A/B on the same atelet build, with and without this change. - Setup: microVM sandbox class (1 vCPU, 1920 MiB guest) on a c3-highmem-192-metal node. - Workload: a recorded 21-step coding-agent trace (SWE-bench astropy-7336, conda-based image), replayed in 4 cycles. - Rounds: 3, alternating between the two builds. Each round runs the trace once with pause and once with suspend/resume between cycles. - Exec time is the wall time to replay one cycle's steps. - In every row, the ranges of the two builds across the rounds don't overlap. | Metric | Without | With | Change | |---|---|---|---| | Exec time, cycle 1 (pause / suspend path) | 2,128 / 2,169 ms | 1,462 / 1,403 ms | −31% / −35% | | Exec time, all 4 cycles (pause / suspend path) | 9,475 / 9,468 ms | 8,457 / 8,360 ms | −11% / −12% | | Resume, median (suspend path) | 1,051 ms | 939 ms | −11% | | rootfs upper, sum over 5 suspends | 79.9 MB | 30.9 MB | −61% | | Last snapshot, stored in GCS | 111.3 MB | 101.8 MB | −9% | Without the change, the actor rewrote 663 of the image's `.pyc` files (11.9 MB). 662 of them hold the same bytecode as the shipped copy, apart from the embedded source path. The other one is a module the trace edits. Cold start from the golden snapshot showed no significant difference. ## Testing - `gofmt`, `go vet`, `golangci-lint` and `go test -race ./internal/imagecache/...` are clean. The new test fails without the fix. - This package's root-gated tests weren't run locally; CI runs them under sudo. - The A/B above ran this change on a microVM cluster. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…ports (agent-substrate#2612) Part of Actor Lifecycle v2 agent-substrate#2069. First PR for agent-substrate#2570. With asynchronous suspend, atelet uploads a suspended actor's snapshot and later detaches the actor from its node on its own timers. This adds the API atelet uses to tell ate-api-server about both. `ReportActorSnapshotState` carries the full snapshot state atelet holds for an actor on its node. The state lists the latest local snapshot, the one being uploaded, and the newest durable one. Each report supersedes older ones. `ActorSnapshotState` and `SnapshotReport` mirror atelet's messages in agent-substrate#2592, since a public proto can't import an internal one. `SnapshotReport` repeats four fields of the public `Snapshot`: `uuid`, `generation`, `fidelity`, and `object`. It doesn't reuse `Snapshot` because that message requires `owner`, which atelet doesn't know. `Snapshot` also carries fields the control plane owns, such as `locality` and the durable copy's status, which a report shouldn't set. This PR adds only the proto, the generated code, and request validation with tests. The RPC returns `Unimplemented` until the handler lands. The following PRs add the function that applies a report to the actor, the `controlapi` method that applies it under the actor's lease, and the `workerservice` handler. Reports will be applied while holding the actor's lease, like any other change to an actor. If another workflow holds the lease, such as a resume, the report is rejected with `Aborted` and atelet retries it. Open question for review: which name do we use, `ReportActorSnapshotState` or `ReportSnapshotStatus`? - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…y pressure (agent-substrate#2030) ## Summary After `RestoreWorkload` loads a MicroVM snapshot into guest memory, the snapshot files (`memory-ranges` and filesystem tarballs) remain cached in the worker pod's cgroup memory—either because staged files in `restore-state/` are cleaned up too late (or not at all for `.tar` archives), or because preserved local checkpoints in local-checkpoint/ keep their page cache resident. Under tight pod memory limits or high density, this leftover page cache overlaps with active guest RAM and subsequent checkpoints, triggering kernel dirty-page throttling (`balance_dirty_pages`) and memory-reclaim stalls. This PR proactively reclaims snapshot page cache during `RestoreWorkload` before resuming the guest, without introducing synchronous disk I/O stalls. ## Changes - **Immediate disposal / cache eviction of restore files before Resume**: Right after guest memory is loaded and filesystem layers are extracted (before client.Resume), temporary single-link files in restore-state/ (memory-ranges, rootfs-upper-*.tar, durable-dir-*.tar) are truncated and removed so the kernel discards their cached pages in-memory without disk writeback. When preserve_restore_dir is set (direct local pause restore) or files are hard-linked (Nlink > 1), the underlying snapshot files are preserved and only clean cached pages are evicted. - **Non-blocking background page-cache eviction**: A lightweight background pass inspects remaining snapshot files (such as preserved `local-checkpoint/` snapshots) via `cachestat` and calls `POSIX_FADV_DONTNEED` strictly when pages are clean (`Dirty == 0 && Writeback == 0`), reclaiming cgroup memory as soon as background flushes finish without blocking on disk writeback. ## Benchmarks Evaluated head-to-head on a 15-node `c3-highcpu-4` GKE cluster comparing `main` vs this PR (`14b5d346`). ### 1. Memory-Constrained Local Pause/Resume (`glutton_1user_512mi_uvm_pause_768mi`) *1 user, 512 MiB active guest RAM (64 MiB dirtied before each pause), 768 MiB worker pod memory limit, 1m duration.* | Metric | Baseline (`main`) | This PR | Delta | | :--- | :---: | :---: | :---: | | **Total Requests / Throughput** | 58 (`1.00 req/s`) | **69 (`1.19 req/s`)** | **+19.0%** | | **`PauseActor` Avg** | 1,522 ms | **756 ms** | **-50.3% (2.0x faster)** | | **`PauseActor` P50 / P75** | 620 ms / 2,600 ms | 780 ms / **910 ms** | +25.8% / **-65.0%** | | **`PauseActor` P90 / P99** | 4,000 ms / 5,000 ms | **970 ms / 1,300 ms** | **-75.8% (4.1x) / -74.0% (3.8x)** | | **`ResumeActor` Avg / P50** | 573 ms / 610 ms | **459 ms / 430 ms** | **-19.9% / -29.5%** | | **`ResumeActor` P90 / P99** | 830 ms / 830 ms | **540 ms / 630 ms** | **-34.9% / -24.1%** | > **Note**: On `main`, `PauseActor` exhibits a bimodal distribution—fast (`~620 ms`) for the first 1–2 cycles while free cgroup memory remains, followed by `2.6s–5.0s` synchronous writeback stalls once un-evicted page cache fills the `768 MiB` cgroup. Evicting the snapshot cache smooths `PauseActor` into a steady `~750–970 ms` distribution, cutting average latency in half (`-50.3%`) and P90 by `4.1x`. ### 2. High-Density GCS Suspend/Resume (`m1_density_512mi_uvm`) *50 concurrent users, 500 actors, 60 workers across 15 nodes (4 vCPU /node), 512 MiB guest RAM, 1 GiB worker memory limit, 5m duration.* | Metric | Baseline (`main`) | This PR | Delta | | :--- | :---: | :---: | :---: | | **Total Requests / Throughput** | 3,676 (`12.29 req/s`) | **3,809 (`12.74 req/s`)** | **+3.6%** | | **Total Failures** | 132 (`3.59%`) | **108 (`2.84%`)** | **-18.2% fewer failures** | | **`ResumeActor` Failure Rate** | 21 / 97 (`21.65%`) | **13 / 112 (`11.61%`)** | **-46.4% failure rate** | | **`ResumeActor` Avg / P75** | 11,757 ms / 12,000 ms | **9,189 ms / 9,700 ms** | **-21.8% / -19.2%** | | **`ResumeActor` P90 / P99** | 36,000 ms / 62,000 ms | **18,000 ms / 48,000 ms** | **-50.0% (2.0x) / -22.6%** | | **`FirstResume` Avg / P90 / P99** | 2,727 ms / 6,400 ms / 30,000 ms | **2,073 ms / 4,100 ms / 20,000 ms** | **-24.0% / -35.9% / -33.3%** | | **`SuspendActor` Failure Rate** | 45 / 518 (`8.69%`) | **35 / 544 (`6.43%`)** | **-26.0% failure rate** | | **`SuspendActor` Avg / P50** | 9,925 ms / 7,600 ms | **9,532 ms / 6,900 ms** | **-4.0% / -9.2%** | ### 3. High-Density Local Pause/Resume (`m1_density_512mi_uvm_pause`) *50 concurrent users, 500 actors, 60 workers across 15 nodes (4 vCPU /node), 512 MiB guest RAM, 1 GiB worker memory limit, 5m duration.* | Metric | Baseline (`main`) | This PR | | :--- | :---: | :---: | | **Aggregated P50** | 230 ms | **230 ms** | | **`ResumeActor` P50 / Avg** | 5,100 ms / 6,601 ms | **4,600 ms (-9.8%)** / 6,804 ms | | **`ResumeActorFirstResume` P50** | 1,400 ms | **1,400 ms** | | **`PauseActor` P50** | 6,800 ms | 7,800 ms | > **Note**: In `m1_density_512mi_uvm_pause`, the `1 GiB` worker pod limit already fits the `512 MiB` guest RAM plus a hard-linked local checkpoint, and unlike GCS `Suspend/Resume` (where single-link staged files are truncated in-memory before writeback), local `Pause` checkpoints must persist to disk—so latency under 50 concurrent users on 4-vCPU nodes is dominated by **node CPU contention and shared disk write bandwidth** rather than page-cache pressure. ## Testing - Unit tests in `cmd/ateom-microvm/snapshotcache_test.go`.
…ate#2585) Component redeploys call `EnsureCRDs`. If any CRD on a hand-kept list was missing, it applied the whole generated directory: every CRD plus ate-controller's RBAC. A component redeploy could then update the existing CRDs and RBAC ahead of ate-controller. The list had also drifted and left out `CSIDriverConfig`. Now: - `EnsureCRDs` creates only the missing CRDs. Existing CRDs and ate-controller's RBAC change only through `DeployCRDs` (`deploy ate-system`, `deploy ate-controller`). - Both read the CRDs from the generated directory, so the hand-kept list is gone. - On a fresh cluster, component commands no longer apply ate-controller's RBAC. Only ate-controller uses it, and `deploy ate-controller` applies it itself. Part of agent-substrate#1686 - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
Move existing hardcoded runsc flags into kodata/flags.json to make it easier to manage and add flags. Flags are reloaded automatically when file changes to allow for setting flags during runtime, e.g. enable debug logs. *AI disclosure: AI tooling assisted in writing and testing parts of this change.* - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. #### Breaking change <!-- If users must act when they upgrade, describe what they must do, and ask for the `breaking-change` label. Otherwise delete this section. What is a breaking change? Something that requires customer code/configuration changes to make their agents work. But more broadly it might include any behavior change that customers were relying on (Hyrum’s law). Examples of breaking changes: - Moving from 1:1 pods to actors to multiple actors in one pod. - Changing actor primary keys or identity formats. - Adding authorization or authentication where none was required before. --> Fixes agent-substrate#2062 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
…benchmarks (agent-substrate#2469) `ate-setup --fake-data-plane` installs the fake data plane from agent-substrate#2300 in place of the real one, so a benchmark's isolate mode can drive ate-api-server and Postgres past what real nodes can restore. This replaces agent-substrate#2366, which grafted the fakes onto a live install from `deploy.sh`. Following the review there, the installer installs the fakes directly and never installs the real atelet. The installer only chooses the data plane; the benchmark tunes fake-atelet. ## Usage ```bash hack/install-ate.sh --deploy-ate-system --fake-data-plane benchmarking/workloads/deploy.sh --deploy --worker-count 200 \ --fake-atelet-env FAKE_ATELET_DELAY=500ms ``` In `tests.yaml`, `isolate: {delay: 500ms}` does both. ## What's in this PR 1. **ate-setup: install the fake data plane with `--fake-data-plane`.** - A `fake-data-plane` kustomize component: - runs fake-atelet in atelet's own DaemonSet, keeping atelet's identity, node selector and storage settings; - scales ate-controller to zero; - adds fake-workersync. - fake-atelet reads its tuning flags from an optional `fake-atelet-tuning` ConfigMap that no install creates or resets. - Only `deploy ate-system` switches between the real and fake data planes: - single-component redeploys refuse a mismatch; - a fake install refuses live worker pods; - a real install removes fake-workersync before ate-controller starts. - fake-workersync: - places Workers on the fake-atelet nodes; - keeps them across restarts; - reports ready only once ate-api-server answers. 2. **benchmarking: run isolate mode on the fake data plane install.** - `deploy.sh --fake-atelet-env` writes the tuning ConfigMap. - A `tests.yaml` `isolate` block selects the mode, and its keys become that tuning. - On the fake data plane, `deploy.sh` waits on the pool's `readyReplicas` instead of a worker Deployment. ## Things to look at - **Release images:** fake-atelet and fake-workersync join `images.Components`, so release builds publish them. - **Switching:** a fake install needs a cluster with no live worker pods, so delete any WorkerPools first. - **ate-controller at zero:** its NetworkPolicy and egress MITM trust controllers stop too. Fake actors need neither. - **User class:** fake actors don't exist, so the mode suits only a user class that sends them no requests, such as `ate_api.py`. ## Testing - Unit tests cover: - the component's render over every bundle; - the switching guards and the fake-workersync removal; - fake-atelet's tuning variables; - fake-workersync's node grace, retry and readiness; - the `isolate` mapping. Addresses agent-substrate#2261 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…gent-substrate#2220) Working on agent-substrate#1563 Fixes: agent-substrate#2149 Lets Substrate's own components call ateapi under `--experimental-enable-authz` without being global owners. | Role (on `global`) | Component | Permissions | |---|---|---| | `controller` | ate-controller | create/list workers; get/update/delete/drain worker | | `egress_gateway` | atenet-egress | get actor; get actor egress policy; list actors (health check) | | `ingress_router` | atenet-router | resume actor; list actors (health check) | - `authz.New(..., authz.SystemPrincipals{...})` adds these as contextual tuples, **for mTLS principals only**, so a JWT whose `sub` matches a component's SPIFFE ID gets nothing. - AccessPolicy can't grant these roles. - ateapi flags `--controller-spiffe-id`, `--egress-gateway-spiffe-id`, `--ingress-router-spiffe-id` default to the install's SPIFFE IDs in ateapi's namespace. Static grants rather than AccessPolicy bindings: they work from first boot, can be bound to mTLS, and can't be dropped by a policy update. **Testing:** interceptor tests per role (allowed and denied, JWT with a component's ID, other SPIFFE IDs), principal validation, AccessPolicy validation, and `fga model test`.
…-substrate#2593) Update the API style guide to reflect the change. Part of agent-substrate#1378
… ci runners (agent-substrate#2625) ... and isn't necessary, it effectively asserts timing constants with an integration test.
…gent-substrate#2425) Atunnel was hardcoded to use CONNECT over H/1. This causes atunnel to open a new mTLS connection to egress gw for each new atunnel connection, causing elevated memory and CPU usage in egress gateway. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Yan Avlasov <yavlasov@google.com>
…ubstrate#2621) the nfs server runs in the cluster, so it dies first and the node's hard nfs mounts retry forever. shut the mounts down before deleting.
…gent-substrate#2521) Part of agent-substrate#587. This is the first part of the [worker sandbox versions design](https://github.com/zoez7/substrate/blob/d72bfc5d48dab71fe19fef31eb586a5c82ba7076/docs/workerpool-sandbox-assets-refresh-en.md), after agent-substrate#2393. agent-substrate#2393 made each WorkerPool record the SandboxConfig it uses in `status.sandboxClasses[].configRef`. This PR is the atelet side of getting that config to the workers. For every worker pod on its node, atelet computes the runtime hash of every enabled version of the pool's config, creates the worker's directory, and writes a versions file in it. atelet keeps the mapping from each hash back to the SandboxConfig version. The worker side is not in this PR. No ateom reads the file yet. Reading it at startup and registering again when the file changes come in the next PR. ## How the syncer works Each worker pod on the node is a key in one workqueue, drained by a single worker. The sections below explain the parts of the diagram that need a reason. ```mermaid flowchart TD subgraph Events["Informer events"] P["Pod Add / Delete"] WP["WorkerPool Add<br/>Update: configRef changed"] SC["SandboxConfig Add<br/>Update: Generation changed"] D["Startup scan of ateoms/ directories"] end P -->|"Add(uid)"| Q[("workqueue<br/>key = pod UID")] WP -->|"Add immediately / Update random delay 0-30s<br/>the pool's local pods"| Q SC -->|"random delay 0-30s<br/>local pods of pools using the config"| Q D -->|"Add, or AddAfter(until 1 minute old)"| Q W["run: wait for Pod, WorkerPool<br/>and SandboxConfig caches to sync"] --> D W --> G Q --> G["run: queue.Get()<br/>single worker"] G --> L{"pod found<br/>in cache?"} L -->|"no"| R["removeDir(uid)"] L -->|"DeletionTimestamp set"| N["skip<br/>wait for Delete event"] L -->|"yes"| S["syncPod"] S --> S1["resolve + plan<br/>pool -> configRef -> versions for this arch"] S1 --> S2["build<br/>hash each version's asset and pause image digests"] S2 --> S3["write<br/>atomically write sandbox-versions.json if changed"] R --> F{"finish(err)"} N --> F S3 --> F S1 -. "error" .-> F S2 -. "default version error" .-> F F -->|"success / errPermanent"| FG["Forget"] F -->|"fewer than 8 retries"| RL["AddRateLimited<br/>doubling from 1s, max 5 min"] F -->|"8 retries reached"| FG RL --> Q ``` ## A worker pod is scheduled to a node atelet watches the worker pods on its own node, the WorkerPools, and the SandboxConfigs. When a worker pod is scheduled to the node, atelet: - Finds the pod's WorkerPool and the config in its `configRef`. It never uses `pendingConfigRef`, which the atecontroller has not confirmed yet. - Picks the config's enabled versions that have assets for the node's architecture. - Computes each version's `sandbox_runtime_hash`: a sha256 over one sorted `key=value` line per asset digest and the pause image digest. Version names and URLs are left out, so the same binaries under another name or in another bucket are the same runtime. The hashes come from the SandboxConfig spec alone, so this downloads nothing. - Creates the worker directory `/var/lib/ate/ateoms/<pod-uid>/` and writes `sandbox-versions.json` in it. For a gVisor pool on an amd64 node using the bundled `gvisor-default`: ``` cat /var/lib/ate/ateoms/<pod-uid>/sandbox-versions.json ``` ```json { "defaultRuntime": { "sandbox_runtime_hash": "sha256:bd071f1eff6bc2b6b6598aaec03540a7c0f86163586282da2e4400a6de072b5d" } } ``` - `defaultRuntime` comes from the config's default version. It is what new and cold-booted actors start under. - `restorableRuntimes` lists the other Enabled versions in spec order. The worker can only restore snapshots under them. It is omitted when there are none. - `RegisterWorkerRequest` reports these two separately, which is why the file keeps them apart instead of a flat list of hashes. - Each runtime is listed once. A version whose hash equals an earlier one's is left out. - A non-default version with no assets for the node's architecture, or with a pause image not pinned by digest, is left out. If the default version has either problem, no file is written, since no worker of the pool could start actors on this node. The file does not promise that a runtime's assets are on the node. The existing prewarmer still downloads each config's default version, and Run/Restore still fetch any version on demand. Fetching through `cmd/atelet/internal/filecache` comes in a follow-up PR (see the TODOs in the code). ## The SandboxConfig changes, or the pool switches config When an operator edits the SandboxConfig's spec, or the pool's `configRef` moves to another config, atelet rewrites the file of every affected worker on the node. Every node sees the change within about a second, and each rewrite will make the worker register again. So that every worker in the fleet does not hit ateapi at once, each atelet waits a random delay of up to 30 seconds before it rewrites. A newly scheduled worker pod is enqueued without that delay, since only its own node acts on it and the worker is waiting for the file. Changes that do not affect the file, such as a SandboxConfig update that leaves its spec unchanged, a `pendingConfigRef`, or a replica count change, do not trigger a rewrite. ## A sync fails - If the pool has no resolved config yet, or the pool or config is not in the cache, atelet keeps the existing file and retries. The 8 retries take about 4 minutes. After that it gives up until the pool's config changes or atelet restarts. - `errPermanent` covers a pod with no worker pool label, and a config whose default version has no assets for the node or has a pause image not pinned by digest. Retrying cannot fix them, so they are not retried. ## atelet starts, the worker pod goes away, or atelet restarts - The syncer never holds up atelet startup. It waits for the Pod, WorkerPool and SandboxConfig caches to sync before it scans the worker directories or drains the queue. Before then, every pod looks gone and no pod's config resolves. While a list/watch fails (e.g. Forbidden while an RBAC rollout lags the binary), files wait and existing ones stay as they are. - When atelet restarts, the informer's initial list replays every pod as an Add, so every worker's file is rewritten from current state. - The startup scan removes directories left by pods that went away while atelet was down. It waits until a directory is a minute old, because its ateom may have created it before the pod reached atelet's cache. - The ateom no longer removes its directory on shutdown, only its own socket. The directory and the file now survive a container restart. ## Other changes - atelet's ClusterRole can now list and watch WorkerPools. - atelet watches only the worker pods on its own node (`spec.nodeName` field selector), so the list/watch cost scales with the pods on the node, not the cluster. - `hardware.AttrSandboxRuntimeHash` (`sandbox_runtime_hash`) replaces the `gvisor_asset_hash` TODO. ## Not in this PR - The ateom reading the file when it starts, and waiting for it if atelet has not written it yet. - The ateom watching the file and registering again when it changes. - Fetching assets through `cmd/atelet/internal/filecache` and removing the prewarmer. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…ING states (agent-substrate#2620) A paused actor no longer has its own state. "Paused" is now defined by the actor's snapshots alone: an actor is paused when it is `ACTOR_STATE_SUSPENDED` and its latest snapshot is local only, that is, it has a `locality` and no `durable_snapshot`. A suspended actor is `ACTOR_STATE_SUSPENDED` with a durable latest snapshot. `ACTOR_STATE_PAUSED` and `ACTOR_STATE_PAUSING` are deleted. This changes how the state is represented, not what the system does. `PauseActor` and `SuspendActor` stay separate workflows and keep sending the same atelet requests. Resume already chose its restore source and node pinning from the snapshot shape (`findLatestLocalSnapshot`, `status.assigned_node`), so it is unchanged apart from its state gate. atelet is untouched. Three commits, one per step: 1. **Pause leaves the actor SUSPENDED with a node-local snapshot.** `FinalizePaused` commits `SUSPENDED` instead of `PAUSED`; `locality`, `assigned_node` and pruning are as before. The workflows that read `PAUSED` now read the shape through three helpers in `snapshot_helpers.go`, all built on `latestSnapshot`: - `latestSnapshotIsLocalOnly`: the paused shape. - `latestSnapshotIsDurable`: the suspended shape (also true with no snapshot at all, which has nothing to upload). - `latestSnapshotIsPendingLocal` (commit 3): a pause in progress. `SuspendActor` returns at once for `SUSPENDED` + durable and uploads the local snapshot for `SUSPENDED` + local only, as it did for `PAUSED`. `PauseActor` is idempotent on the paused shape. `RevertActor` accepts the paused shape and still refuses a durable `SUSPENDED` actor. Changing an actor's template still refuses a paused actor. `ResumeActor`'s state gate accepts `SUSPENDED` only. 2. **Delete `ACTOR_STATE_PAUSED`.** Enum value, `ate.actor.state` metric label `paused`, and the remaining mentions. 3. **Delete `ACTOR_STATE_PAUSING`.** A pause in progress is `SUSPENDING` like a suspend in progress; the latest snapshot tells them apart (a pause has minted a local snapshot with no locality yet and no durable copy, a suspend has a pending durable copy). The pause workflow re-enters on that shape, `SuspendActor` refuses an actor mid-pause as it refused a `PAUSING` one, and a crash during a pause is still attributed to the pause operation. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…gent-substrate#2583) ### What A built-in agent-session script, `personal-assistant-light`: the personal-assistant day (61 wakes, heartbeats, crons, briefing, compaction, nightly sweep) with only the RAM arrays scaled down, from 960Mi to 28Mi. Files, reads, CPU burn, dwell and think gaps are unchanged, so each turn's I/O and the wake pattern are identical to personal-assistant. Plus a README paragraph. ### Why personal-assistant models a gateway with a 1.2 to 1.5 GiB heap, so every pause writes a memory image near 1 GB. Runs that ask how Substrate behaves when an agent's own state is tens of MB (actors per node, cheap pauses, fast restores) had no script with the always-on wake pattern. ### Memory image per pause | | gateway start | mid-morning onward | |---|---|---| | gVisor | about 40 MB | 80 to 95 MB | | micro-VM, actor memory limit 512Mi | about 140 MB | up to about 200 MB | | micro-VM, actor memory limit 256Mi | about 125 MB | about 125 MB | Under gVisor the image is the workload; the Go heap grows to about twice the live arrays over a day of turns. Under micro-VM the image is the guest's populated pages: about 100 MB of kernel, agent and page cache before the script writes anything, bounded by the actor's memory limit. The micro-VM rootfs tar adds up to 22 MB. `min_actor_memory` is 256Mi; the script fits a 128 MiB guest. ### Testing `TestEmbeddedScriptsAreValid` loads the new file. `hack/verify/boilerplate.sh` passes. Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>
EItanya
force-pushed
the
sync/upstream-main-20261014
branch
from
October 14, 2026 13:28
d12c865 to
8448b47
Compare
EItanya
marked this pull request as draft
October 14, 2026 13:28
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261014
branch
from
October 14, 2026 13:31
8448b47 to
9ab50aa
Compare
EItanya
marked this pull request as ready for review
October 14, 2026 15:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges 42 upstream commits through aba56bb71, including snapshot-based pause state, per-worker sandbox asset preparation, API cleanup, HTTP/2 egress tunnels, and control-plane network policies.
Resolves the atelet startup and egress fixture conflicts while preserving the fork's local-cluster mode and agentgateway behavior. Updates the Helm health probes, WorkerPool watch permissions, component identities, and ingress policies to match the new control plane. Adds chart coverage and updates the credential-provider manifest test for the separate health listener. Adapts the networking checks to agentgateway: policy-denial probes trust the gateway CA, and CONNECT evidence must identify the correct actor and destination alongside the verified HTTPS response.
Breaking change
ListOptions,MintActorJwt, the renamed protobuf types and timestamps, andWakeupProbe. Paused actors now useSUSPENDEDplus node-local snapshot state instead ofPAUSED/PAUSING.ate.dev/api-access=true. The chart now denies ingress to the control-plane namespace by default; custom workloads there need their own policies. Custom health probes must use port 9091 instead of the metrics port.Please apply the
breaking-changelabel.The branch contains one merge commit above the fork main, retaining the upstream commits as ancestors.
Validation on
9ab50aa5a: all four CI checks pass, including unit/race tests, root verification, Helm verification, and both E2E workflows. The two previously failing networking tests pass in the Helm gVisor suite and both runtime lanes of the main workflow.Agentgateway E2E: 276 tests reported across gVisor and microVM, 8 skipped, no failures.
Helm E2E: 174 tests reported across gVisor, microVM, credential injection, and identity rotation, 6 skipped, no failures.
Local chart tests (13), chart lint, manifest rendering, and CRD parity passed. Local full verification and the final networking test rerun hit the 20 GiB temporary filesystem limit; CI completed those checks on the submitted revision.
Unit and chart tests pass
Upgrade documentation updated
Please merge with a merge commit to preserve upstream ancestry for the next sync.