ateom: export telemetry through an atelet unix-socket OTLP relay by chw120 · Pull Request #809 · agent-substrate/substrate

Chenyi Wang (chw120) · 2026-08-08T01:36:15Z

ateom runs inside the worker pod that hosts the actor, and exported OTLP straight to the collector over the pod's network. This adds a node-local relay: ateom pushes OTLP/gRPC over a unix socket that atelet serves and forwards to the collector, so a worker pod needs no network path of its own to export spans and metrics.

Four things motivate it:

Blast radius. The pod runs untrusted agent code, so allowing it egress to the collector makes the collector reachable to anything that escapes the sandbox. A unix socket cannot leave the node.
Connection count. Worker pods are heavily oversubscribed; N ateoms per node each held their own collector connection. They collapse into atelet's single per-node one.
Interference. ateom transparently redirects actor egress to its own atunnel listener, and its own outbound traffic has to stay clear of the rules it installs. A unix socket is not IP traffic.
Shutdown loss. Teardown frees the actor's network and then the pod goes away, which is when the spans describing teardown are still queued in the batch processor. atelet outlives the worker pod.

The relay forwards the OTLP request verbatim rather than decoding and re-exporting, so each ateom's own resource (service.name, service.instance.id) survives instead of being absorbed into atelet's.

It is best-effort: an ateom that finds no socket at startup logs it and exports directly to OTEL_EXPORTER_OTLP_ENDPOINT as before, so this is a no-op for a cluster running an older atelet. atelet likewise declines to serve a relay when no collector is configured, since it would accept spans only to drop them. Both halves stay off with --otlp-relay-socket="".

The socket lives in ateompath.BasePath, the hostPath already mounted at the same path into atelet and into every ateom pod, so no new volume or controller change is needed.

Also includes:

End-to-end tests covering the full serverboot-to-collector path.
Observability documentation updates for Jaeger tracing.

It's a good idea to open an issue first for discussion.

Tests pass
Appropriate changes to documentation are included in the PR

ateom runs inside the worker pod that hosts the actor, and exported OTLP straight to the collector over the pod's network. This adds a node-local relay: ateom pushes OTLP/gRPC over a unix socket that atelet serves and forwards to the collector, so a worker pod needs no network path of its own to export spans and metrics. Four things motivate it: - Blast radius. The pod runs untrusted agent code, so allowing it egress to the collector makes the collector reachable to anything that escapes the sandbox. A unix socket cannot leave the node. - Connection count. Worker pods are heavily oversubscribed; N ateoms per node each held their own collector connection. They collapse into atelet's single per-node one. - Interference. ateom transparently redirects actor egress to its own atunnel listener, and its own outbound traffic has to stay clear of the rules it installs. A unix socket is not IP traffic. - Shutdown loss. Teardown frees the actor's network and then the pod goes away, which is when the spans describing teardown are still queued in the batch processor. atelet outlives the worker pod. The relay forwards the OTLP request verbatim rather than decoding and re-exporting, so each ateom's own resource (service.name, service.instance.id) survives instead of being absorbed into atelet's. It is best-effort: an ateom that finds no socket at startup logs it and exports directly to OTEL_EXPORTER_OTLP_ENDPOINT as before, so this is a no-op for a cluster running an older atelet. atelet likewise declines to serve a relay when no collector is configured, since it would accept spans only to drop them. Both halves stay off with --otlp-relay-socket="". The socket lives in ateompath.BasePath, the hostPath already mounted at the same path into atelet and into every ateom pod, so no new volume or controller change is needed. Also includes: - End-to-end tests covering the full serverboot-to-collector path. - Observability documentation updates for Jaeger tracing.

Chenyi Wang (chw120) · 2026-08-08T01:42:11Z

Benjamin Elder (@BenTheElder) FYI, this is the PR for collecting ateom traces. Please review it when you have time. Thank you.

Benjamin Elder (BenTheElder)

Human vetted, AI first pass.

Benjamin Elder (BenTheElder) · 2026-08-08T02:25:47Z

+	}
+	// A socket left behind by a previous atelet would make Listen fail with
+	// EADDRINUSE even though nothing holds it.
+	if err := os.RemoveAll(s.sockPath); err != nil {


🤖 should-fix 🟡 – RemoveAll is a lot more than this needs, and the blast radius on a misconfigured path is the node's whole ateom state. The comment explains the goal as clearing a stale socket so Listen does not hit EADDRINUSE — os.Remove does exactly that and fails on a directory, which is the right answer. RemoveAll instead succeeds: --otlp-relay-socket=/var/lib/ateom-gvisor (the BasePath itself, an easy slip since the socket lives under it) would have atelet silently delete the image cache, every actor bundle, the staged runsc binaries and the local checkpoints, at startup, before anything else runs.

Stop already uses os.Remove on the same path, so the two ends of the socket's lifecycle disagree about which call is appropriate.

Thanks Benjamin Elder (@BenTheElder) for review. Updated the code to use os.Remove.

Benjamin Elder (BenTheElder) · 2026-08-08T02:25:47Z

+	// gRPC resolves a "unix://" target to a unix socket dialer natively, so the
+	// OTLP exporters above this connection are unchanged: OTLP is gRPC, and gRPC
+	// needs only a reliable byte stream.
+	conn, err := grpc.NewClient("unix://"+sockPath,


🤖 nit 🟢 – This only builds a valid target for an absolute sockPath. With a relative one, unix://otlp.sock parses otlp.sock as the authority with an empty path, so the dial fails — and it fails lazily, at first export, which is precisely the outcome the os.Stat check above exists to avoid. Stat would have passed, since a relative path resolves fine against the working directory, so the deterministic startup fallback is bypassed and the telemetry is quietly lost instead.

grpc.NewClient("unix:"+sockPath, ...) — the single-colon form — takes everything after the colon as the path and handles both shapes, or the flag could reject a non-absolute path up front.

Added the validation.

Krisztian F (krisztianfekete)

I'd hold off on this. It's the ateom part of the same relay we're designing in #761. That relay has to rewrite telemetry identity, and it needs a network listener regardless since a unix socket won't work with microvm guests.

Can we design that end first and let ateom telemetry leverage that, I am fine with the socket itself, but let's not codify stuff we are about to undo. cc. Da Huang (@git286)

Da Huang (git286) · 2026-08-10T14:39:21Z

I'd hold off on this. It's the ateom part of the same relay we're designing in #761. That relay has to rewrite telemetry identity, and it needs a network listener regardless since a unix socket won't work with microvm guests.

Can we design that end first and let ateom telemetry leverage that, I am fine with the socket itself, but let's not codify stuff we are about to undo. cc. Da Huang (Da Huang (@git286))

Krisztian F (@krisztianfekete) I think these are two different hops.

#761 is actor → ateom/atelet: untrusted client inside the sandbox, identity has to be rewritten, and it needs a network listener because a unix socket is invisible to a micro-VM guest. This PR is ateom → atelet: ateom is a normal container in the worker pod with the BasePath hostPath mounted, whatever runtime it drives, and its identity is already correct. The socket will work for ateom -> atelet no matter it's gvisor or microvm, right?

Krisztian F (krisztianfekete) · 2026-08-10T15:17:32Z

I'd hold off on this. It's the ateom part of the same relay we're designing in #761. That relay has to rewrite telemetry identity, and it needs a network listener regardless since a unix socket won't work with microvm guests.
Can we design that end first and let ateom telemetry leverage that, I am fine with the socket itself, but let's not codify stuff we are about to undo. cc. Da Huang (Da Huang (Da Huang (@git286)))

Krisztian F (Krisztian F (@krisztianfekete)) I think these are two different hops.

#761 is actor → ateom/atelet: untrusted client inside the sandbox, identity has to be rewritten, and it needs a network listener because a unix socket is invisible to a micro-VM guest. This PR is ateom → atelet: ateom is a normal container in the worker pod with the BasePath hostPath mounted, whatever runtime it drives, and its identity is already correct. The socket will work for ateom -> atelet no matter it's gvisor or microvm, right?

You're right, I take back the microvm concern!

The one thing I'd still ask for is scoping. Forwarding as-is is the correct behaviour for ateom telemetry, but right now that covers relay itself rather than about this particular traffic. Whether #761 rewrites at ateom or at atelet, this relay is on this path too, so it will eventually has traffic that has to be modified alongside traffic that must not be. Could we limit this to ateom source now, so adding the rewriting path later is an extension rather than a redoing it?

internal/otlprelay imports the OTLP collector protos directly, but go.mod still carried the module as indirect: go mod tidy was never rerun after the relay landed, so hack/verify/go-modules.sh regenerates, finds a diff and fails run-tests. Nothing else moves. vendor/modules.txt already marked the module explicit, which is why every local build and test stayed green and only CI caught it.

Addresses review on agent-substrate#809: Krisztian's scoping request and Benjamin's two bugs. Scope the relay to ateom sources. Forwarding a request untouched is not a property of relaying in general; it is correct only for a source whose resource is already correct. ateom's is, so the relay now says so and refuses everything else with PermissionDenied, a resource declaring no service.name included. The case this excludes is actor telemetry: actors share a hostname ("runsc") and an interior IP, so their series merge unless identity is injected from outside the actor (agent-substrate#761), and rewriting in flight is the negation of pass-through. The two do not share this socket today -- agent-substrate#761 is actor->ateom and this relay is ateom->atelet -- but naming the contract while it still holds makes that path an added branch later rather than a re-argument about whether pass-through was ever safe. A batch is refused whole rather than having the offending resource dropped, since a partial success the sender reads as success loses telemetry silently. The allowlist duplicates the serviceName constants in cmd/ateom-*, which are package main and cannot be imported, so a test reads them back out of the source: a typo there would otherwise refuse every real ateom export while the package's own tests, sharing the typo, kept passing. Remove the stale socket with os.Remove rather than os.RemoveAll. The path comes from a flag and lives in BasePath, the host directory that also holds every ateom's own socket, so a value naming the directory would have had RemoveAll empty it. os.Remove refuses a populated directory, turning that into a startup error instead of data loss. Reject a relative socket path in both halves. gRPC does not resolve one: "unix://foo/r.sock" parses as authority "foo" and path "/r.sock". Because grpc.NewClient is lazy, that wrong target was accepted at startup and failed per export afterwards with the spans already gone, defeating the existence check in Dial that makes the direct-export fallback a startup decision. atelet and ateom would also resolve a relative path against different working directories, so it cannot name one socket for both halves. An absent socket still falls back quietly; only a malformed path errors.

Chenyi Wang (chw120) · 2026-08-11T00:01:19Z

I'd hold off on this. It's the ateom part of the same relay we're designing in #761. That relay has to rewrite telemetry identity, and it needs a network listener regardless since a unix socket won't work with microvm guests.
Can we design that end first and let ateom telemetry leverage that, I am fine with the socket itself, but let's not codify stuff we are about to undo. cc. Da Huang (Da Huang (Da Huang (Da Huang (@git286))))

Krisztian F (Krisztian F (Krisztian F (@krisztianfekete))) I think these are two different hops.
#761 is actor → ateom/atelet: untrusted client inside the sandbox, identity has to be rewritten, and it needs a network listener because a unix socket is invisible to a micro-VM guest. This PR is ateom → atelet: ateom is a normal container in the worker pod with the BasePath hostPath mounted, whatever runtime it drives, and its identity is already correct. The socket will work for ateom -> atelet no matter it's gvisor or microvm, right?

You're right, I take back the microvm concern!

The one thing I'd still ask for is scoping. Forwarding as-is is the correct behaviour for ateom telemetry, but right now that covers relay itself rather than about this particular traffic. Whether #761 rewrites at ateom or at atelet, this relay is on this path too, so it will eventually has traffic that has to be modified alongside traffic that must not be. Could we limit this to ateom source now, so adding the rewriting path later is an extension rather than a redoing it?

Thanks Krisztian F (@krisztianfekete) Da Huang (@git286) for review. Updated the code to scope it to only allow ateom sources, "ateom-gvisor" and "ateom-microvm".

Tim Bai (baizhenyu)

The overall change looks good to me, left some minor comments

Tim Bai (baizhenyu) · 2026-08-11T01:34:52Z

+// injected from outside the actor (#761) -- a rewrite, not a forward. Nothing
+// but ateom reaches this socket today; naming the contract now makes that path
+// an added branch later rather than a re-argument about pass-through.
+var ateomServices = map[string]bool{


The allowlist is a good contract, but worth saying explicitly in this comment that it is not a security boundary: service.name is written by the client, so a compromised process in a worker pod can claim ateom-gvisor and pass the check. It prevents accidental misuse (an actor SDK pointed at the socket) and keeps the pass-through contract explicit for #761 — that's its real value. If we ever need to authenticate the peer, that takes per-pod sockets or UDS peer credentials (SO_PEERCRED), which also connects to the open identity question on #741.

Tim Bai (baizhenyu) · 2026-08-11T01:34:52Z

+	}
+	if _, err := os.Stat(sockPath); err != nil {
+		if errors.Is(err, fs.ErrNotExist) {
+			slog.WarnContext(ctx, "OTLP relay socket absent, exporting telemetry directly over the pod network",


The fallback is decided once, at startup, and only surfaces as a per-pod log line. Two consequences worth tracking:

A worker that boots while atelet is restarting (e.g. a DaemonSet rollout) exports directly over the pod network for its whole life.

While any worker can fall back, network policy cannot deny worker-pod egress to the collector — so the blast-radius motivation in the PR description only becomes true in a phase 2 (periodic re-probe, or a strict relay-only mode, plus removing the controller's endpoint injection).

Could we add a metric (or resource attribute) that marks a worker as on the fallback path, so the migration is observable fleet-wide? And a short note in the docs section that the egress lockdown is future work, so nobody reads this PR as already delivering it.

Tim Bai (baizhenyu) · 2026-08-11T01:34:52Z

+	// Lazy by design: grpc.NewClient does not block on the collector being up,
+	// so atelet startup does not depend on the collector's readiness.
+	upstream, err := grpc.NewClient(target,
+		grpc.WithTransportCredentials(insecure.NewCredentials()))


Rejecting https:// instead of silently downgrading (line 308) is the right call and more honest than serverboot's current behavior. But note this collides with #741: when WithInsecure() is removed from serverboot so operators can run TLS collectors, the relay becomes the remaining plaintext-only hop, and every ateom on the node is behind it. Worth a follow-up (or a checkbox on #741) to give the relay's upstream leg TLS support at the same time.

Tim Bai (baizhenyu) · 2026-08-11T01:34:52Z

+//	go test ./internal/otlprelay/ -run TestEndToEndThroughServerboot -v
+func TestEndToEndThroughServerboot(t *testing.T) {
+	sink, collector := startFakeCollector(t)
+	sock := startRelay(t, collector)


The isolation this test's comment claims is weaker than stated. startRelay does t.Setenv(endpointEnv, collector), and collector is a directly routable loopback host:port — so the endpoint variable does not point at the fake collector "through the relay only". An exporter that ignored ExporterConn would read that env and try to deliver directly.

Today a bypass still fails the test, but only by accident: the value is a bare host:port, url.Parse rejects it (scheme can't start with a digit), the SDK discards the env var and dials the default localhost:4317, and the sink times out. That protection evaporates if the SDK's parsing changes or if this env ever gains an http:// scheme.

One-line fix: after startRelay (the relay resolves its upstream once, in NewServer, so this doesn't affect it), re-point the env at a parseable but unroutable address before InitTracing:

t.Setenv(endpointEnv, "http://127.0.0.1:1")

Then a bypassed relay fails loudly and deterministically. Note the replacement must parse as a URL — an unparseable value gets ignored and falls back to localhost:4317, where a dev machine might run a real collector.

Krisztian F (krisztianfekete) · 2026-08-11T15:28:32Z

+	// directly for its whole life, so the socket should exist before any worker
+	// pod on this node boots.
+	if relay, err := otlprelay.NewServer(ctx, *otlpRelaySocket); err != nil {
+		serverboot.Fatal(ctx, "Failed to create the OTLP relay", err)


Are we sure about this?
Everything else here degrades nicely, so Fatal might be too much. What do you think?

Krisztian F (krisztianfekete) · 2026-08-11T15:33:32Z

+	// socketMode keeps the relay socket reachable by the ateom pods on the node
+	// (which do not necessarily share atelet's uid) while staying off-node by
+	// construction. The socket is inside BasePath, a root-owned host directory.
+	socketMode = 0o666


We get root from the controller, aren't we?
Would 0600 be enough here?

Krisztian F (krisztianfekete) · 2026-08-11T15:39:43Z

+// The relay forwards the OTLP request message verbatim rather than decoding it
+// into SDK records and re-exporting. Pass-through keeps each ateom's own
+// resource (service.name, service.instance.id, pod attributes) intact, so its
+// spans stay attributed to ateom instead of being absorbed into atelet's. That
+// holds only for a source whose resource is already right, so the relay carries
+// ateom telemetry only; see ateomServices.


Could we word the guarantee as being about the ateom source, so the rewriting path is an extension later rather than a contradiction?

Krisztian F (krisztianfekete) · 2026-08-11T15:42:08Z


+### The ateom OTLP relay
+
+ateom is the one component that does not talk to the collector itself. It exports over a unix socket at `/var/lib/ateom-gvisor/atelet-otlp.sock`, which `atelet` serves and forwards to the collector on the node's network ([`internal/otlprelay`](../internal/otlprelay)):


As per this comment: #761 (comment), by Da Huang (@git286) the GKE pipelines needs checking as after this change ateom telemetry will be passing trough atelet connections, so k8sattributes might break.

Krisztian F (krisztianfekete) · 2026-08-11T15:45:44Z

+		if specific == "" {
+			continue
+		}
+		if resolved != "" && resolved != generic && specific != resolved {


resolved != generic is false on the first non-empty pass, so the guard can only ever fire when both signal-specific vars are set.

Krisztian F (krisztianfekete) · 2026-08-11T15:48:26Z

+			return nil, err
+		}
+	}
+	return t.upstream.Export(ctx, req)


gRPC won't pass the incoming metadata along to the upstream call, so if an ateom has OTEL_EXPORTER_OTLP_HEADERS set the headers will be dropped so we cannot do auth I guess?

Same applies for metrics below as well.

Krisztian F (krisztianfekete) · 2026-08-11T15:50:51Z

+	// Lazy by design: grpc.NewClient does not block on the collector being up,
+	// so atelet startup does not depend on the collector's readiness.


What happens when the collector is down?

Can we set compression here?

Benjamin Elder (BenTheElder) reviewed Aug 8, 2026

View reviewed changes

Max Smythe (maxsmythe) added the area/observability label Aug 8, 2026

Julian Gutierrez Oschmann (juli4n) requested review from Jeff Luo (JeffLuoo) and Da Huang (git286) August 8, 2026 15:14

Da Huang (git286) requested review from Tim Bai (baizhenyu) and removed request for Jeff Luo (JeffLuoo) August 10, 2026 13:06

Krisztian F (krisztianfekete) suggested changes Aug 10, 2026

View reviewed changes

Benjamin Elder (BenTheElder) assigned Benjamin Elder (BenTheElder) and Bowei Du (bowei) Aug 10, 2026

Chenyi Wang (chw120) added 2 commits August 10, 2026 16:43

Tim Bai (baizhenyu) reviewed Aug 11, 2026

View reviewed changes

Da Huang (git286) mentioned this pull request Aug 11, 2026

[Design] Actor Otel Telemetry Relay #853

Open

Krisztian F (krisztianfekete) reviewed Aug 11, 2026

View reviewed changes


		### The ateom OTLP relay

		ateom is the one component that does not talk to the collector itself. It exports over a unix socket at `/var/lib/ateom-gvisor/atelet-otlp.sock`, which `atelet` serves and forwards to the collector on the node's network ([`internal/otlprelay`](../internal/otlprelay)):

		// Lazy by design: grpc.NewClient does not block on the collector being up,
		// so atelet startup does not depend on the collector's readiness.

Uh oh!

Conversation

Chenyi Wang (chw120) commented Aug 8, 2026

Uh oh!

Chenyi Wang (chw120) commented Aug 8, 2026

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Krisztian F (krisztianfekete) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Da Huang (git286) commented Aug 10, 2026

Uh oh!

Krisztian F (krisztianfekete) commented Aug 10, 2026

Uh oh!

Chenyi Wang (chw120) commented Aug 11, 2026

Uh oh!

Tim Bai (baizhenyu) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Krisztian F (krisztianfekete) left a comment •

edited

Loading