Repository navigation
Sync upstream DNS enforcement and sandbox runtime updates - #78
Merged
Merged
Conversation
…cs (agent-substrate#2576) `otelgrpc` sets `server.address` on `rpc.client.call.duration` from the remote address of the client connection: `atelet` dials each `ateom` over a Unix socket path containing the worker pod UID, and TCP clients record the IP address of an individual server pod. This produced one `server.address` value per pod, causing metric series cardinality to grow with worker pod count and churn. * Add a metric `View` in `serverboot.newMeterProvider` that strips `server.address` from all instruments in the `otelgrpc` scope while keeping `server.port` (one value per service). * Remove `server.address` from `docs/metrics/registry/metrics.yaml` and `docs/metrics/substrate.yaml`, and document the view on `rpc.client.call.duration`. Fixes agent-substrate#2556 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…2055) This prevents actors being scheduled on a worker that is restarting or temporarily unreachable. - [ x ] Tests pass - [ x ] Appropriate changes to documentation are included in the PR
…compatible workers (agent-substrate#2536) Restoring a sandboxed actor's memory checkpoint (`MEMORY` snapshot fidelity) requires compatibility between the worker that captured the checkpoint and the worker restoring it — across the sandbox class (`gvisor`, `microvm`), the host hardware (`architecture`, CPU features), and the sandbox runtime assets (`gvisor_asset_hash`, microVM kernel/rootfs digests). Fixes agent-substrate#2249 agent-substrate#2250 Following agent-substrate#2381 (where workers began self-reporting their `default_runtime` and `restorable_runtimes` during registration), this PR closes the control-plane loop across checkpointing, scheduling, and restore: 1. **Assignment-Pinned Runtime Stamping**: When an actor is placed on a worker, the control plane records the specific `SandboxRuntime` that will serve that actor on `WorkerAssignment.sandbox_runtime`. Subsequent checkpoints taken during that assignment (`Pause`, `Suspend`, `Tag`) stamp `Snapshot.sandbox_runtime` from the assignment rather than re-reading the worker's live `default_runtime`. This guarantees that if a worker updates its `default_runtime` while an actor is still running under an older runtime version, checkpoints taken from that actor are still stamped with the runtime that actually produced the snapshot. 2. **Runtime-Aware Scheduling with Graceful Fallback**: When resuming an actor from a `MEMORY` snapshot under its current template, the scheduler prefers workers whose `default_runtime` or `restorable_runtimes` match the snapshot's `sandbox_runtime` (with `hardware.Matches` returning `false` if either runtime is `nil`). If no runtime-compatible worker has capacity (and the actor is not pinned to a local pause checkpoint), the scheduler relaxes the runtime constraint so the actor can still make progress on any otherwise-eligible worker. 3. **Consistent Restore Fidelity Downgrade (`MEMORY` -> `VOLUMES`)**: At restore time, compatibility is evaluated directly against the persisted `WorkerAssignment.sandbox_runtime` — the same field that will stamp the next checkpoint — keeping the restore decision and future snapshot stamps in lockstep across retries and re-entries. When the assigned runtime does not match the snapshot's required runtime, all durable snapshots (actor-owned, explicit `source_tag` clones, and borrowed golden snapshots) uniformly downgrade the `RestoreRequest` fidelity to `SNAPSHOT_FIDELITY_VOLUMES` and log a warning. --- ### Scenarios #### 1. Cold Boot (No Snapshot) or Non-Memory (`VOLUMES` / `ROOTFS`) / Repointed Snapshot - **Scheduling**: No memory checkpoint is being restored (`RequiredRuntime == nil`), so any active worker matching the template's `SandboxClass`, selectors, and resource limits is eligible. - **Assignment**: `WorkerAssignment.sandbox_runtime` is set to the bound worker's `default_runtime`. - **Restore & Next Checkpoint**: The actor cold-boots via `Run` (or restores `VOLUMES`/`ROOTFS`), and any subsequent pause/suspend stamps the new snapshot with `WorkerAssignment.sandbox_runtime`. #### 2. Warm Restore on a Matching Worker (`default_runtime` or `restorable_runtimes`) - **Scheduling**: The actor has a stamped `MEMORY` snapshot (taken under the current template). The scheduler filters for workers where either `default_runtime` or an entry in `restorable_runtimes` matches the snapshot's `sandbox_runtime`. - **Assignment**: Under the worker's row lock, `WorkerAssignment.sandbox_runtime` records the exact runtime entry on that worker that matched the snapshot (`default_runtime` on a steady-state worker, or the matched `restorable_runtimes` entry during a runtime rollout). - **Restore & Next Checkpoint**: Because `WorkerAssignment.sandbox_runtime` matches the snapshot's required runtime, `atelet.Restore` runs at full `SNAPSHOT_FIDELITY_MEMORY`. If the actor was restored via a `restorable_runtimes` entry on a worker whose `default_runtime` has rolled forward, its next pause or suspend is still stamped with the matched `restorable_runtimes` entry. #### 3. Durable Snapshot Fallback When Compatible Workers Are Full (Actor-Owned, `source_tag` Clone, or Borrowed Golden Snapshot) - **Scheduling**: The initial `Schedule` pass with `RequiredRuntime` finds no capacity (`ErrNoCapacity`). Because the snapshot is durable (not a node-local pause checkpoint), the workflow relaxes `RequiredRuntime = nil` and re-runs `Schedule`, picking an eligible worker running a different runtime. - **Assignment**: `WorkerAssignment.sandbox_runtime` records the fallback worker's `default_runtime` (or, if a compatible worker freed up between the two `Schedule` calls, the matched runtime from the unrelaxed snapshot requirement). - **Restore & Next Checkpoint**: `ensureAteletRestored` detects that `WorkerAssignment.sandbox_runtime` does not match the snapshot's required runtime, logs a warning, and downgrades the `RestoreRequest` fidelity to `SNAPSHOT_FIDELITY_VOLUMES` so the actor boots a fresh sandbox under the new runtime while preserving its volume state. Subsequent checkpoints are stamped with the new worker's runtime. #### 4. Local Pause Checkpoint on an Incompatible Node - **Scheduling**: A `PAUSED` actor holds a node-local checkpoint on `AssignedNode`. Because nothing releases the node-local checkpoint and claim on fallback yet, `canRelaxRuntime` returns `false`. If no worker on `AssignedNode` supports the snapshot's `sandbox_runtime`, scheduling returns `ResourceExhausted` rather than silently stranding the local checkpoint. #### 5. Concurrent Worker Re-registration & Workflow Re-entry Races - **Stale cache vs. row-locked worker**: If `atelet` re-registers a worker and drops an older runtime from `restorable_runtimes` before the scheduler's watch cache catches up, `BindActorToWorker` evaluates admission under the worker's DB row lock and `assignWorkerAttempt` computes `WorkerAssignment.sandbox_runtime` from that fresh row-locked worker rather than the cached copy. - **Re-entry in `RESUMING`**: If `ResumeActor` fails after persisting `RESUMING` (e.g., during volume attachment) and re-enters later, `validateAssignedWorker` accepts a worker assigned via runtime relaxation, and `ensureAteletRestored` derives `runtimeIncompatible` from the persisted `WorkerAssignment.sandbox_runtime` so the restore fidelity and the next checkpoint's stamp cannot drift even if the worker's advertised runtimes changed in the interim. ### Testing & Docs - **Functional test** (`TestResumeActor_SandboxRuntimePlacementAndFallback`): Exercises stamping, compatible worker placement, full-worker fallback to `SNAPSHOT_FIDELITY_VOLUMES` (for both actor-owned and borrowed golden snapshots), re-stamping on subsequent suspend, and `restorable_runtimes` warm restore + `PauseActor` -> `SuspendActor`-from-`PAUSED`. - **Cluster E2E test** (`internal/e2e/suites/sandboxruntime` — `TestSnapshotRuntimeStamp`): Verifies live worker `DefaultRuntime` stamping across `ResumeActor`, `PauseActor` (local checkpoint + warm restore), `SuspendActor`-from-`PAUSED`, `SuspendActor`-from-`RUNNING`, and `SourceTag` actor clones. - **API Guide** (`docs/api-guide.md`): Documents `Snapshot.sandbox_runtime` and `WorkerAssignment.sandbox_runtime` placement and fallback semantics. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ubstrate#2330) The `runsc create`, `start`, and `restore` commands for the pause container previously wrote their stdout/stderr to the shared worker pod log without actor attribution. `RunWorkload` and `RestoreWorkload` now forward that CLI output and the pause container's stdio through the existing actor log envelope, labeled with the actor and `ate.actor.container.name="_pause"`. This covers cold startup and VOLUMES/MEMORY restore. JSON fields are preserved, and reserved `ate.*` fields cannot override platform attribution. `kubectl ate logs actors` excludes `_pause` by default, keeping the pause container's runsc startup banners and host paths out of the default application log view. Application container logs and actor lifecycle records remain visible. Pass `-c _pause` to inspect the pause container's runsc diagnostics. The attributed records remain in the worker pod log and centralized logging backends. Restore requests reject unsupported snapshot fidelity values before acquiring actor locks or setting up networking and log pipes. Both workload paths call `StartJSONLogPipe` directly, consistent with application containers. The RPC closes its pipe writer when it returns; the forwarding goroutine does not retain the RPC context. This change attributes runsc CLI/pause stdio output. With runsc debug logging disabled, sentry and gofer stdio is directed to `/dev/null`; this change does not enable that logging. `--alsologtostderr` uses glog text by default. Tests cover that text format and runsc JSON diagnostics using `msg`, `level`, and `time`. Addresses the pause-container invocation portion of agent-substrate#2271 and the default log display concern raised in review. One-shot runsc helpers, sentry panic capture, and the micro-VM implementation remain separate work. No new registry attributes are introduced. Validation after merging main `f03327bd`: - Linux/amd64, Go 1.27.2: full `go test -race -count=1 ./...` passed for 152 packages, with PostgreSQL/envtest integration enabled by `CI=true`. - Pause attribution, fs checkpoint/restore arguments, fidelity rejection, and CLI source/container filtering regressions passed 20 repetitions with the race detector. - All 15 repository verification scripts passed. The pinned golangci-lint v2.13.2 used Go 1.27.1 and a 15-minute timeout after its Go 1.27.2 export-data incompatibility was confirmed. The earlier package-loading timeouts are retained in the validation record. Both VOLUMES and MEMORY restore paths retain main's fs restore arguments while forwarding pause output through the actor log pipe. The checkout remained clean. Verification ran as an unprivileged user. Root-gated mount tests and real gVisor E2E were not run for this head. AI assistance was used to resolve the merge conflicts and run verification.
Fixes agent-substrate#2075 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
With serverName empty, VerifyConnection skipped the hostname check and accepted any certificate from pod-identity CA. Require it via CEL rule on CSIDriverTLSConfig and fail closed in resolveTLSConfig #### Breaking change - CRD: CEL rule on `CSIDriverTLSConfig` requires a non-empty `serverName` when `enabled` is true. Manifest regenerated. - Runtime: `resolveTLSConfig` returns an error when `serverName` is empty, so configs stored before the rule fail closed instead of connecting unverified. - Docs: `serverName` is documented as required when TLS is enabled. Fixes agent-substrate#2079 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…-system's default-deny ingress (agent-substrate#2637) Since agent-substrate#2522, ate-system denies ingress by default and admits API clients only from namespaces labeled `ate.dev/api-access=true`. The `benchmarking` namespace that the automation's runner Job templates create had no labels, so on a fresh cluster the runner could not dial ate-api-server. The boomer worker died ensuring its atespace (`CreateAtespace ... dial tcp ...:443: i/o timeout`), and the locust master waited for a worker that never connected until the Job timed out. This labels the namespace in both runner Job templates (`runner-job.yaml.tmpl` and `nighthawk-ingress-runner-job.yaml.tmpl`), as `benchmarking/locust/deploy.sh` already does for manual runs. `RunnerNamespaceTest` reads the label that the `ate-api-server` NetworkPolicy selects on and checks both templates against it, so the two can't drift apart. It fails on both templates without the change. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…; actor JWT claims move from ate.dev to actor.ate.dev (agent-substrate#2523) Part of agent-substrate#777 The object-store plugin's control sidecar will exchange an actor JWT for a downscoped object-store credential scoped to that actor. atelet needs a way to obtain that JWT over the channel it already authenticates on, which is `WorkerService`, where it already mints the certificate counterpart with `MintAteomActorCertificate`. This adds `MintAteomActorJwt` beside it. Nothing calls it yet. atelet starts to in a later PR, together with the node plugin exchange. Two commits: 1. **Proto.** `MintAteomActorJwtRequest` mirrors the certificate request (actor, UID guard against a deleted-and-recreated actor) plus `audiences` from `Control.MintActorJwt`, each non-empty. The lifetime is fixed at one hour, so there is no expiration field. Response is the token and its expiry. Generated Go and validation code, and the validation wrapper. 2. **Handler.** The caller must be atelet and the actor must exist. The mint is then refused unless the caller's node hosts the actor, as `RequestActorSuspend` already requires. The actor's state decides which status field is trusted, since both can outlive the hosting they record: the node of the assigned worker while the actor is resuming, running or suspending, and `assignedNode` while it is suspended or suspending with no worker and its latest generation is a local checkpoint with no durable copy, which is how a paused actor looks now that `PAUSED` is no longer a state. The whole rule lives in a new `controlapi.HostingNode`, in `hosting.go`, so the suspend and resume workflows' own notion of a paused actor is what it reads, and so the certificate RPC can reuse it in its follow-up. Anything else resolves to no node and is refused: a worker gone from the store, a reverting actor whose assignment outlives its released worker, a crashed or deleting actor whose `assignedNode` is only a record of its last node, and a suspended actor with a stale `assignedNode`. The node check runs before the UID check and every refusal is `NotFound`, so a node learns nothing about actors it does not host, not even that one exists. A UID mismatch is then answered `Aborted`, as `MintAteomActorCertificate` answers it. Each handler composes its own `actoridjwt.Claims` and signs with the actor-id JWT pool, as `Control.MintActorJwt` does. The two tokens differ only in kind: a token minted here speaks for an ateom acting on the actor's behalf, with subject `ateom-for-actor/<atespace>/<name>` and its claims under `ateom-for-actor.ate.dev`, while `Control.MintActorJwt` mints the actor's own token with subject `actor/<atespace>/<name>` and claims under `actor.ate.dev`. The zero kind is rejected, so a token cannot be minted without choosing. A relying party can therefore tell the actor from a Substrate component acting for it, and the snapshot credential exchange will accept only the `ateom-for-actor` kind, so actor code cannot reach its own snapshots. Follow-up, in its own PR: `MintAteomActorCertificate` gets the same host check. It is an existing RPC on the actor boot and resume path, so the change deserves its own review and e2e run. #### Breaking change Actor JWTs no longer carry the `ate.dev` claim. The actor's own token now carries the same fields under `actor.ate.dev`, and the new ateom-for-actor token carries them under `ateom-for-actor.ate.dev`. The OIDC discovery document's `claims_supported` lists the two new names. Any relying party that reads `ate.dev` from an actor JWT, agentgateway included, has to read `actor.ate.dev` instead, and must not accept an `ateom-for-actor` token as the actor. Relying parties should accept both `ate.dev` and `actor.ate.dev` before the API server is upgraded, and drop `ate.dev` an hour after it, once every token minted before the upgrade has expired. Please add the `breaking-change` label. - [x] Tests pass with `go test -race ./...`. New unit tests cover the handler and the claims package: claim contents per kind, `exp` equal to `expire_time`, the host check for a foreign node, a foreign node with a stale UID, a paused actor on each node, a paused actor being suspended, a crashed actor on its last node, an unhosted actor, a lost worker, and each validation and lookup error. - [x] Appropriate changes to documentation are included in the PR. The RPC, messages and both tokens' claims are documented in the proto, and no guide lists WorkerService RPCs.
…1989) Fixes agent-substrate#1395 `/readyz` now fails while the store is unreachable, so the replica leaves the Service. `/healthz` is unchanged. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…bstrate#2656) This avoids needing network ingress policy overrides. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR Signed-off-by: Yan Avlasov <yavlasov@google.com>
…gent-substrate#2618) - Follow-up to agent-substrate#2531: --upstream-connect now defaults to true, so the Envoy router reaches atunnel through mTLS CONNECT tunnels on 8443. --upstream-connect=false goes back to HTTPS on 443. - An empty --upstream-credential-bundle now needs --upstream-connect=false, and the startup error says so. The check only applies to the Envoy ingress router; the egress gateway and agentgateway don't use the flag (agentgateway always uses CONNECT). - Split xDS server setup out of startEnvoyDataplane so a new test can parse real flags and check the built snapshot for both settings. - The e2e gRPC downgrade test now expects only gRPC Unavailable, so it fails if the router falls back to the HTTPS path. - Updated architecture.md, threat-model.md and the router manifest comment. Related: agent-substrate#1208 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Rewrites the upgrade runbook for an `in-place` roll by K8s deployment. Each component is upgraded where it runs, in order: podcertificate-controller, atelet, the workers, ate-api-server, the CRDs and ate-controller, atenet, then the default SandboxConfig. ate-controller follows the API server because it is a client of it, and new workers start from the pod template the old ate-controller renders. The old runbook ran the old and new dataplane side by side instead. This deliberately gives up two things the old runbook had. The first is zero loss. Workers are replaced through the eviction path: `SIGTERM`, then 30 minutes for the actor to be suspended before `SIGKILL`. Actors already need to handle this for node upgrades and preemption. An actor that is suspended in time loses nothing. One that isn't crashes and loses the work since its last snapshot. The old runbook avoided this by having the operator suspend every actor on a node before moving it. The second is instant rollback. Rollback is the same roll in reverse, from the old release, instead of flipping a node label back. Keeping both versions running needs spare capacity and the version label. ## TODOs Some places are marked TODO because they depend on unfinished work: 1. [DONE] The install record `ate-setup` writes (agent-substrate#1887, implemented in agent-substrate#2223) 2. The compatibility policy and the version skew policy. The runbook links both, while both poclies are still in design. 3. Per-release SandboxConfig names > Part of agent-substrate#1686 Merges together with agent-substrate#1812
…te#2454) The `apitool` tool used to guess the resource model from names. It kept a hard-coded list of resource names and assigned each RPC to whichever one appeared longest in the method name. That list is stale and subresource methods were therefore attributed to their parent. For example, `GetActorEgressPolicy` counted as an Actor method and `ListWorkerActorAssignments` as a `Worker` method. Because of that, subresources and singletons were linted against the wrong resource or not at all. This change records the resource model in the proto itself and makes apitool read it from there. Changes The PR has three commits, which can be reviewed one at a time: 1. Add proto options to annotate API resources and methods. 2. Annotate the Control API's resources and methods. 3. Simplify the apitool CLI to read resources from the proto annotations. Lint rule changes: New: - `standard-method-naming`: the verb, then for a subresource one of its parents (or Global), then the resource. List uses the plural. - `subresource-naming`: a subresource's message is named without its parent type. - `singleton-no-list`: a singleton has no List method. - `custom-response-name-matches-method`: a custom method returns {MethodName}Response, not the resource. Changed: - `get-request-single-objectref` is now get-request-shape, and it covers subresources: the parent's ObjectRef plus name. - The List, Create, Update and Delete shape rules understand subresources, singletons, Global parents and scope. For example, a List request has an atespace field only for an atespace-scoped top-level resource.
…t-substrate#2586) #### Breaking change An ActorTemplate's `sandboxConfig.sandboxClass` is now a string, `gvisor` or `microvm`, the same values the WorkerPool and SandboxConfig CRDs use. The `SANDBOX_CLASS_GVISOR` and `SANDBOX_CLASS_MICROVM` spellings are rejected. To upgrade: - In ActorTemplate manifests and API requests, replace `SANDBOX_CLASS_GVISOR` with `gvisor` and `SANDBOX_CLASS_MICROVM` with `microvm`. - Go clients of `pkg/proto/ateapipb`: the `SandboxClass` enum is removed and `SandboxConfig.SandboxClass` is a `string`. - Existing ActorTemplates read back with an empty `sandboxClass`, because the stored enum value does not decode into the new string field. Templates are immutable, so delete them and their actors before upgrading and recreate them afterwards. - The `SANDBOX CLASS` column of `kubectl ate get actortemplates` now prints `gvisor` or `microvm`. Fixes agent-substrate#2406
…ive_actors (agent-substrate#2221) Part of agent-substrate#1590 Benchmark runs now record: - Per-operation RPS and latency over time (Locust `--csv-full-history`), not just the aggregate. - Restore and checkpoint latency from the ate_actor_{restore,checkpoint}_duration histograms (total phase, every snapshot kind) - `active_actors` (p50/p95/avg + timeseries) in `server_summary.json`. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…Internal (agent-substrate#2614) Fixes agent-substrate#2613 On GKE with a GCS backend, the object-store plugin's Workload Identity token gets `412 PreconditionFailed` ("The operation requires that Uniform Bucket Level Access be enabled.") from the public `gs://gvisor`. `toStatus` maps that, like any error it doesn't recognize, to `Internal`, and atelet didn't fall back on `Internal`, so sandbox creation failed after agent-substrate#2439. **Changes** - atelet falls back to an anonymous read on `Internal`. In practice that's any unexpected plugin failure. That's acceptable because anonymous reads only reach public objects, and atelet still checks the sha256. - So a broken plugin isn't hidden: the plugin logs a scrubbed warning when it returns `Internal`, and atelet warns when it falls back. - Updated the `AssetProvider` contract, `api-guide.md` and threat model T-08. **Testing** - Unit tests: `go test ./cmd/atelet/... ./internal/objectstoreplugin/... ./pkg/objectstorage/...` - GKE 1.37 (Envoy dataplane): reproduced the 412 on `main`. With this PR, gVisor prewarm falls back and succeeds, and the counter demo passes create, persistence across requests, suspend, and resume from snapshot. A follow-up issue will discuss the long-term design for plugin fallback. --------- Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>
Part of agent-substrate#2069 actor lifecycle v2 If a local snapshot exists for the requested uuid, restore from it instead of downloading from object storage. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
Move the platform-neutral DNS relay test helper out of the Linux-only file so Darwin tests compile.
fixes agent-substrate#2180 1) ateom now runs under a minimal init (internal/ateominit) when it would be PID 1. the init reaps orphans like the runsc sandbox, so ateom no longer calls wait4(-1) and no longer races os/exec for subprocess exit statuses. this drops internal/childreap and the RunCommand gating, which is what kept the checkpointed sentry a zombie while runsc delete ran. 2) after a MEMORY checkpoint ateom-gvisor waits for the sandbox to exit and be reaped (pidfd opened before the checkpoint, so pid reuse can't fool it) before running cleanup. This PR was AI assisted. I have reviewed the changes.
Fixes agent-substrate#2619 atunnel waited for the gateway's CONNECT answer for as long as the actor stayed active. A gateway at its connection cap queues the CONNECT; the actor's client gives up in seconds, but the queued CONNECT kept its slot and later opened a tunnel nobody wanted. The tunnel open now has a 10s deadline on the Egress side. Since the HTTP/2 multiplexing change, ending the dial context already resets the CONNECT stream, which withdraws it from the gateway's queue; this PR adds the test that pins that behavior and the deadline that triggers it for an actor that has gone away. A tunnel already returned is unaffected. The HTTP/1 fallback path is left as it was. Tested with `go test -race -count=20 ./internal/atunnel/...` and the `hack/verify` scripts. New test: an Egress with a shortened timeout, dialing through the real client to a gateway that never answers, closes the actor's connection at the deadline while the gateway sees the stream reset. Prepared with help from an AI coding agent; reviewed and verified by the author.
…gent-substrate#2562) Working on agent-substrate#1563. Fixes agent-substrate#1696 ### Summary - **Fail-closed interceptor for unregistered RPCs:** `UnaryServerInterceptor` now returns `PermissionDenied` when an RPC is not in `defaultRPCPermissions` while enforcement is enabled. `WorkerService` RPCs, which authenticate `atelet` in their handlers via `ateletauth`, are explicitly registered with `inHandler(...)`. - **Exhaustive coverage test (`TestDefaultRPCPermissions_CoversEveryServedRPC`):** Walks every method of every service in `ateapi.proto` and fails if any RPC has neither a permission rule nor an `inHandler` reason, if the registry names an RPC that isn't in `ateapi.proto`, or if a service gains a streaming RPC (which the unary interceptor would not check). - **Tags (`Tag` type and rules):** Models tags as atespace children (`can_get`: `viewer`, `can_update`/`can_delete`: `editor` on the parent atespace; `can_create_tag`: `editor` on the source actor's atespace; `can_list_tags`: `viewer` on the atespace or `global:root`). Calling `CreateActor` with an explicit non-golden `source_tag` requires `can_use_tag` (`editor` on the tag's atespace), regardless of whether the tag's scope is `TAG_SCOPE_ATESPACE` or `TAG_SCOPE_PUBLISHED` (the handler continues to enforce that `TAG_SCOPE_ATESPACE` tags are only used within the same atespace). - **Actor identity credentials:** Restricts `MintActorJwt` and `MintActorCertificate` to `egress_gateway` so an atespace `editor` cannot impersonate an actor. - **Model cleanup:** Removes `global.can_get` and `actor_template.can_update`, which no RPC checks.
…at Worker holds the actor (agent-substrate#2668) Follow-up to agent-substrate#2523. Suspend (and pause), crash, and a resume that finds its Worker ineligible release the actor's Worker before they rewrite its status. In that window, or for good if the status write fails, the status still names the Worker, so its node could mint a one-hour ateom-for-actor JWT, even after it took another actor. - `controlapi.HostingNode` returns a Worker's node only while the store's worker assignment binds that Worker to the actor's UID. A failed lookup refuses the mint. - A paused actor, which has no Worker, still resolves to `assigned_node`; that branch is unchanged. - Real-PostgreSQL tests drive each workflow into that window and through a rebook. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
#### Breaking change `ExternalVolume.status` is gone. Use `storage_volume_id` (empty until the volume is created) and the actor state instead. Please add the `breaking-change` label. Fixes agent-substrate#2414 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Bring the fork up to date with upstream while retaining agentgateway and Helm integration. Wire the DNS policy control client into atelet's existing transport setup and synchronize controller flags and CSI validation in the charts. Align egress test hostnames and policy refresh fixtures with DNS enforcement so requests reach the gateway assertions. Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261014-2
branch
from
October 15, 2026 12:53
8945c4f to
cd1fb39
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge upstream through
2c9a05f48into the fork, bringing in 24 commits covering DNS egress enforcement, runtime-compatible snapshot restore, worker availability, API authorization, and store-aware readiness.The atelet conflict keeps the fork's transport setup and adds upstream's Control client to its existing credential service. The Helm chart now passes the required DNS enforcement flag and includes the updated CSI TLS validation. Agentgateway remains the chart's dataplane.
The credential test uses the service's fully qualified hostname so its DNS query matches the egress policy. The passthrough refresh test keeps DNS allowed while revoking TLS access, so its denial assertion reaches the gateway.
Breaking change
sandboxConfig.sandboxClassvalues fromSANDBOX_CLASS_GVISOR/SANDBOX_CLASS_MICROVMtogvisor/microvm, and update clients for the new protos, including removal ofExternalVolume.status.REFUSED.actor.ate.devinstead ofate.dev. Ateom tokens use the separateateom-for-actor.ate.devclaim.spec.tls.serverNameon CSIDriverConfigs with TLS enabled, and check access-policy grants for the newly authorized RPCs and tag operations.Please apply the
breaking-changelabel.Validation:
hack/verify-all.sh, Helm lint, manifest rendering, CRD parity, and the API-tool race tests passed. The rootgo test -race ./...run passed 156 packages; its only failure was the unchangedinternal/e2ecolor-formatting tests under this shell'sNO_COLOR=1. That package passed withenv -u NO_COLOR go test -race ./internal/e2e. Runtime E2E passed in pr-workflow and helm-e2e. The standard suite ran 141 tests per runtime, with 6 skipped on gVisor and 2 on microVM. Helm's credential-injection and trust-rotation lanes also passed. All PR checks, including unit tests and verifiers, passed on8945c4f92.The branch contains one merge commit,
cd1fb3958, with both histories preserved. Its tree is identical to the revision that passed CI; GitHub is rerunning checks after consolidation.Merge this PR with a merge commit to preserve upstream ancestry for the next sync.