[Design] Actor Otel Telemetry Relay

[git286]

Design: per-actor otel telemetry relay

Status: draft / for discussion
Related: #761 (actor instrumented metrics issue), #503 (telemetry continuity), #450 (lifecycle hooks)

---

1. Summary

Actors cannot correctly identify their own telemetry, so all telemetry from an ActorTemplate collapses into a single time series with many writers.

This proposes that ateom expose a fixed OTLP endpoint inside the worker pod, stamp trusted ate.dev/* identity onto everything that arrives.

The principle is one Substrate already applies to logs:

Attribution belongs to the channel, not the payload.

internal/actorlog/logger.go:132-158 already does this for stdout — atelet injects ate.dev/actor_atespace, actor_name, actor_uid,
actor_template_namespace, actor_template_name, container_name into every log line, out of band. The actor never touches them, so they can't be frozen, forged, or lost. Metrics and traces are the signals that haven't caught up.

It's also the same division of labour Kubernetes already uses. Under Prometheus, identity comes from the scrape target, not the workload — instance and podare assigned by the scraper, and emitting them yourself is an anti-pattern.

Actors can't be scraped, so the relay is that job relocated to somewhere it can still happen. This is not a novel mechanism; it's the conventional one adapted to push.

2. Goals

1. Every actor's telemetry is attributed to that actor, with no cooperation from actor code beyond pointing at an endpoint.
2. Attribution is trusted — derived by the platform, not asserted by the workload — so it can back per-tenant views, quota, and billing later.
3. Attribution survives suspend/resume/migration with nothing re-derived.
4. Storage cardinality is bounded and independent of actor count.
5. Works on both runtimes, gVisor and micro-VM.

3. Non-goals

• Buffered-telemetry loss on onPause: Data. That is OpenTelemetry Application Telemetry Data Loss and Delivery Delays Across Actor Lifecycle Events #503 and needs Actor lifecycle hooks #450's PreSuspend. Complementary, not a substitute.
• Control-plane component telemetry, which works today.
• Per-actor metric series in long-term storage — explicitly rejected

4. Architecture

┌─ worker pod ────────────────────────────────────────┐
│                                                     │
│  ┌─ actor sandbox ─────────┐      ┌─ ateom ───────┐ │
│  │  app                    │      │  identity     │ │
│  │   │ OTLP                │      │  from the     │ │
│  │   └─► 169.254.17.1:4317 ┼──────┼─►activation   │ │
│  │  (no identity in        │      │  receiver     │ │
│  │   payload)              │      │   · stamps    │ │
│  │                         │      │   · separates │ │
│  └─────────────────────────┘      │   · bounds    │ │
│                                   └───────┬───────┘ │
└───────────────────────────────────────────┼─────────┘
                                            │ OTLP, bounded labels,
                                            │ over a unix socket
                                            ▼
                                   ┌─ atelet (DaemonSet) ─┐
                                   │  relays verbatim,    │
                                   │  one egress per node │
                                   └──────────┬───────────┘
                                              ▼
                                   cluster collector → backend

The actor's exporter config is a constant address, so it survives the golden snapshot intact. What answers at that address is per-actor, re-created on every activation. Substrate already relies on this trick twice: the constant interior IP (cmd/ateom-microvm/net.go:28-30 — "Because that address is a CONSTANT, a restored guest's frozen network config stays valid on any pod") and the constant identity path /run/ate/actor-id.

5. Design

5.1 Transport

169.254.17.1:4317 — the existing worker-side veth gateway, already created per activation by both runtimes.

Chosen over a Unix socket in the identity dir because:

• It works on micro-VM. cmd/ateom-microvm/spec.go:91-96 records that the identity bind mount is dropped entirely there — "The micro-VM guest can't see host paths" — so a UDS is unavailable. But kata wires the guest to the same veth via virtio-net (cmd/ateom-microvm/net.go:26-30), so a network endpoint works on both runtimes with no new plumbing.
• No host-fd exposure. runsc gates host Unix sockets behind --host-uds, which cmd/ateom-gvisor/runsc.go:84-96 does not pass. Enabling it would widen the sandbox surface and need a threat-model review. A netstack connection needs neither.
• Checkpoint safety. An open host fd may block runsc checkpoint; a netstack socket is ordinary severed-connection handling.

The cost is that a shared address relies on one-actor-per-worker for unambiguous attribution. That holds today by definition (docs/glossary.md:30-32) and is the thing to revisit if concurrency lands

5.2 Actor-side configuration

OTEL_EXPORTER_OTLP_ENDPOINT=http://169.254.17.1:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta

Env is safe here because these are constants. The golden-snapshot hazard applies to values that must differ per actor; these do not, so freezing them freezes the right answer.

Note these are captured into the golden, so changing them on an existing template requires a golden rebuild (docs/api-guide.md:101: values are inherited "until the golden snapshot is recreated").

Actors emit no identity attributes. Anything they do emit under ate.dev/is overridden.

5.3 Attribution

ateom sets, overriding any existing value:

service.instance.id              (= actor UID; the standard name for this)
ate.dev/actor_uid
ate.dev/actor_name
ate.dev/actor_atespace
ate.dev/actor_template_namespace
ate.dev/actor_template_name
ate.dev/container_name

That is the set for traces and logs. Metrics get a strict subset (§5.4): actor_uid, actor_name, service.instance.id and actor_atespace are all dropped, because all four are unbounded. The signals still join — on the smaller set — and an exemplar walks from a metric datapoint to a span carrying the rest.

Overriding rather than merging is the point: a workload must not be able to attribute its telemetry to another actor or another atespace.

5.4 Cardinality policy

Signal	Actor identity	Rationale
Metrics	stripped before forwarding	a series is durable: index entry, interned labels, chunks, retained for the full window after the actor dies
Traces	kept	a span is an event, not a series — 10k actors/day is 10k rows, not 200k index entries
Logs	kept	already the case today

Forwarded metric labels: actor_template_namespace, actor_template_name, container_name. Output series count is templates × containers × metrics — constant, and independent of how many actors ever ran.

5.5 Forwarding and enrichment

Forwarded telemetry must not be re-enriched with k8s.pod.name / k8s.node.name upstream. Those describe the current host, change on every migration, and are what makes the failure read like ordinary pod churn. ateom should stamp ate.dev/worker_pod / ate.dev/worker_node instead — clearly named as host facts, not workload identity — and keep them off metric resources entirely.

ateom does not dial the cluster collector. It forwards over a node-scoped unix socket served by atelet, which holds the single upstream connection for the node. A direct path requires the worker pod to have network egress to the collector, and the worker pod is the pod running untrusted actor code. Forwarding over a unix socket on a host directory the pod already mounts means the worker pod can be denied network egress entirely and telemetry still flows. No configuration of a direct path yields that property.

Related: #809

5.6 Suspend and resume

Nothing to re-derive. The frozen exporter config points at a constant address; on resume that address is served by the new worker's ateom, which knows the new activation. The gRPC client sees a dead connection and reconnects to the same address, which now leads somewhere else. That is the whole mechanism.

This is why attribution does not depend on #450, though #450 is still needed for flushing (#503).

5.7 Temporality

Delta is required, not preferred.

Under cumulative, ateom would have to hold each actor's last-seen value and compute the merged total itself. On migration the destination ateom has no history and the merged series drops — the original bug, one layer up. Under delta each export is self-contained, so contributions from different workers simply add.

Delta also cancels the golden baseline. The SDK's last-collection bookmark is snapshotted alongside the counter, so a restored actor reports only its own work.

Delta is genuinely lossier. Cumulative self-heals across a dropped export; delta does not, so an onPause: Data kill loses that window permanently. Still the right trade, because what cumulative preserves faithfully is a wrong number.

6. Open questions

1. Concurrency. A shared worker-pod address relies on one-actor-per-worker. If that changes, attribution needs per-actor addressing or a per-actor socket. Worth knowing the cost before it's forced.
2. Gauges: Delta temporality does not apply to gauges. Two actors of one template writing the same gauge now share one series and race on last value. ateom cannot fix this. Maybe we should just make it explicit that per-actor (instance) gauges do not work for substrate and workload should emit logs/events when per-actor gauges are needed, and prefer counters and histograms wherever the quantity is additive.
3. Actor telemetry shares ateom's channel to atelet: Identities do not get mixed up, because atelet forwards messages unchanged and OTLP keeps a separate resource block per sender. The problems are elsewhere. First, if the socket is missing, ateom: export telemetry through an atelet unix-socket OTLP relay #809 falls back to sending over the pod network; that is right for ateom's own data but wrong for actor data, since this design uses the socket so the worker pod needs no network at all. Second, one connection per node carries both kinds, so a noisy actor can delay Substrate's own telemetry. Suggested order: stop the fallback for actor telemetry and rate-limit it in ateom now, then give atelet a second connection to the collector, and only later give actor telemetry its own socket — and only when atelet itself must treat the two kinds differently. Note that a separate socket only tells the two kinds apart; it does not prove who sent the data unless the file permissions change too.

[git286]

cc Krisztian F (@krisztianfekete) Tim Bai (@baizhenyu) Jeff Luo (@JeffLuoo)

[JeffLuoo]

Overall the concept LGTMs, just some comments we need to follow up on:

• 169.254.17.1:4317 won't work in the proposal. The nftables redirect intercepts the OTLP connection before ateom sees it.

ActorEgressRedirectRule (internal/ateomnet/net.go:360-373) matches the source IP 169.254.17.2 and TCP. It has no destination match. The table goes into the worker pod netns, because net.go:571 is outside the NetNSDo at net.go:564, and the chain hooks prerouting. As the result, an actor connection to 169.254.17.1:4317 becomes a connection to 169.254.17.1:<atunnel egress port>. The receiver gets nothing.

The failure is also conditional. The redirect exists only when egress_gateway is set (net.go:363-365, ateom.proto:99-100). An actor with direct egress works. An actor with tunneled egress fails.

We may need to discuss it with people who work on networking layer to review this section and provide guidance on the section.

• container_name works for logs because each container has its own pipeline using WrapContainerLogs in the logger. If we require users to use OTLP in their telemetry pipeline for containers in actor, it has no equivalent so we need to let users configure to set up container name from application layer.

[git286]

Overall the concept LGTMs, just some comments we need to follow up on:

• 169.254.17.1:4317 won't work in the proposal. The nftables redirect intercepts the OTLP connection before ateom sees it.

ActorEgressRedirectRule (internal/ateomnet/net.go:360-373) matches the source IP 169.254.17.2 and TCP. It has no destination match. The table goes into the worker pod netns, because net.go:571 is outside the NetNSDo at net.go:564, and the chain hooks prerouting. As the result, an actor connection to 169.254.17.1:4317 becomes a connection to 169.254.17.1:<atunnel egress port>. The receiver gets nothing.

The failure is also conditional. The redirect exists only when egress_gateway is set (net.go:363-365, ateom.proto:99-100). An actor with direct egress works. An actor with tunneled egress fails.

We may need to discuss it with people who work on networking layer to review this section and provide guidance on the section.

• container_name works for logs because each container has its own pipeline using WrapContainerLogs in the logger. If we require users to use OTLP in their telemetry pipeline for containers in actor, it has no equivalent so we need to let users configure to set up container name from application layer.

Bowei Du (@bowei) Lior Lieberman (@LiorLieberman) haiyanmeng Can you provide suggestion as the network team: under tunneled egress, is 169.254.17.1 supposed to be reachable from the actor at all, or is the gateway address treated as "outside" like everything else?

[krisztianfekete]

Thanks Da Huang (@git286) for putting this together! Adding some comments/questions in-line below (it's very challenging to do this in GH issues, maybe we can move this to Google Docs?):

It's also the same division of labour Kubernetes already uses. Under Prometheus, identity comes from the scrape target, not the workload — instance and pod are assigned by the scraper, and emitting them yourself is an anti-pattern.

The reason per-target series are fine under Prometheus is not just that the scraper assigns identity, it is that instance is the writer identity. Relocating the scraper's job means relocating that part too.

This proposes that ateom expose a fixed OTLP endpoint inside the worker pod, stamp trusted ate.dev/* identity onto everything that arrives.

ate.dev/* is the wrong namespace, we should use internal/ateattr.

Attribution is trusted — derived by the platform, not asserted by the workload — so it can back per-tenant views, quota, and billing later.

Why are we dropping actor_atespace in 5.3 then? Quota and billing cannot work with sampled data. Tenants are not actors; cardinality here is tenant count, which is what we'd need for multi-tenancy, which is in scope, right? Or, usage is calculated as unsampled events like we discussed in #174.

It works on micro-VM. cmd/ateom-microvm/spec.go:91-96 records that the identity bind mount is dropped entirely there — "The micro-VM guest can't see host paths" — so a UDS is unavailable. But kata wires the guest to the same veth via virtio-net (cmd/ateom-microvm/net.go:26-30), so a network endpoint works on both runtimes with no new plumbing.

I've been checking this as part of another issue and think it should work!

Chosen over a Unix socket in the identity dir because:

We should serve 4318/http-protobuf as well as 4317/grpc. The OTel spec's default protocol is http/protobuf, several SDKs ignore or only partially implement OTEL_EXPORTER_OTLP_PROTOCOL, and an actor that defaults to 4318 gets a silent black hole with no error anywhere. Every collector should serve both.

OTEL_EXPORTER_OTLP_ENDPOINT=http://169.254.17.1:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta

Let's also add OTEL_METRIC_EXPORT_INTERVAL. SDK default is 60s, so an actor that lives for shorter periods cannot export and that's very common in this project. We already hit this in #634.

I'd also add OTEL_SERVICE_NAME, or better, default it at the relay. Without it the SDK emits unknown_service: and every actor of every template collapses into one service. Defaulting at the relay is probably better here because it is then fixable without a golden rebuild.

Note that the temporality preference is a request, not an enforcement. A workload can ignore it and some SDKs do not implement it unfortuntately, so the relay must handle cumulative arrivals regardless.

Overriding rather than merging is the point: a workload must not be able to attribute its telemetry to another actor or another atespace.

This might deserve its own design, but roughly we probably have to

1. do this for every attribute, not just the resource. Resource, scope, span, span event, span link, datapoint, log record as a forged ate.actor.uid on a datapoint can work around a resource-level override, and if we strip actor identity from metrics it becomes the only identity attribute reaching storage. Exemplars should also have to be included, once we have them.
2. remove everything under ate.* unconditionally at every phase, then set the trusted values. Same goes for for k8s.* as well I guess,
3. think about some mehanism to make forging very hard. Telemetry identity is the service.namespace + service.name + service.instance.id, so we can have ateom set service.namespace as a trusted override probably.
4. have a trusted resource attribute (ate.telemetry.plane=actor, or similar) for routing. E.g. for the resource_attribute for k8sattributes' pod_association, rate-limiting, other use-cases?

Metrics get a strict subset (§5.4): actor_uid, actor_name, service.instance.id and actor_atespace are all dropped, because all four are unbounded. The signals still join — on the smaller set — and an exemplar walks from a metric datapoint to a span carrying the rest.

See my comments above for atespace. Also, joining via exemplars can only work if the actor also sets them. These are fine limitatations, but we have document them, and choose the approach we want for usage use-case.

Forwarded metric labels: actor_template_namespace, actor_template_name, container_name. Output series count is templates × containers × metrics — constant, and independent of how many actors ever ran.

This is a multi-writer otlp stream that is an anti-pattern because we won't be able to distinguish the emitters.
I'd add k8s.pod.name (+ probably k8s.pod.uid) of the worker pod. This way we have single writers only and cardinality is worker pods not actors, which should be fine.

We also could ship everything here, and do the plumbing/trimming in an OTel collector, but I'd do this once we hit scaling limits with the worker approach above.

Forwarded telemetry must not be re-enriched with k8s.pod.name / k8s.node.name upstream. Those describe the current host, change on every migration, and are what makes the failure read like ordinary pod churn. ateom should stamp ate.dev/worker_pod / ate.dev/worker_node instead — clearly named as host facts, not workload identity — and keep them off metric resources entirely.

k8s.pod.name and k8s.node.name are the semconv registry attributes for these concepts, we can likely use k8sattributes processor via resource_attribute-based pod_association rule. See my comment above about getting rid of these.

Delta also cancels the golden baseline. The SDK's last-collection bookmark is snapshotted alongside the counter, so a restored actor reports only its own work.

Not sure if this is true, I think start time will be also in golden. We probably have to rewrite it in the relay.

Open questions

Concurrency

Agreed, we can revisit it later.

Gauges

+1, with wo additions:

1. asynchronous UpDownCounters can be delta and backends handle them unevenly, so name them too
2. if the relay ever aggregates (see my point above as an alternative), gauges are partially recoverable with min/max/avg rather than last-value-wins. Worth making this a stated limitation in the docs, not just an open question, because it is a real constraint on what workloads can instrument.

Actor telemetry shares ateom's channel to atelet; on the ordering part

I'd be happy if ateom rate limit and the fallback removal would land with the feature, so I would move both out of open questions and into the design.

container_name works for logs because each container has its own pipeline using WrapContainerLogs in the logger. If we require users to use OTLP in their telemetry pipeline for containers in actor, it has no equivalent so we need to let users configure to set up container name from application layer.

I think we can auto-set this via something like OTEL_RESOURCE_ATTRIBUTES=container.name=<name>, if we have access to the name in our OCI layer.