imagecache: add ate.imagecache.requests hit and miss telemetry (#831) - #834
Open
Jeff Luo (JeffLuoo) wants to merge 1 commit into
Open
imagecache: add ate.imagecache.requests hit and miss telemetry (#831)#834Jeff Luo (JeffLuoo) wants to merge 1 commit into
Jeff Luo (JeffLuoo) wants to merge 1 commit into
Conversation
Jeff Luo (JeffLuoo)
force-pushed
the
imagecache-requests-metric
branch
2 times, most recently
from
August 10, 2026 20:47
d5887fa to
5ad0822
Compare
Collaborator
Author
|
Also cc Krisztian F (@krisztianfekete) to review it as well. |
Jeff Luo (JeffLuoo)
force-pushed
the
imagecache-requests-metric
branch
2 times, most recently
from
August 11, 2026 14:13
c174674 to
f34f9d5
Compare
Da Huang (git286)
previously approved these changes
Aug 11, 2026
Jeff Luo (JeffLuoo)
force-pushed
the
imagecache-requests-metric
branch
4 times, most recently
from
August 11, 2026 15:55
b2c1e3e to
985efe6
Compare
Jeff Luo (JeffLuoo)
requested review from
Da Huang (git286) and
Krisztian F (krisztianfekete)
August 11, 2026 16:00
…-substrate#831) The node-local OCI image cache sent no telemetry. The cache is on the Resume path: atelet calls `Store.EnsureImage` before it makes the OCI bundle for an actor. A hit costs no network I/O. A miss causes a pull and an unpack, which takes tens of seconds for a large image. Thus the hit ratio of a node is a leading indicator of Resume latency, but an operator could not see it. Add the `ate.imagecache.requests` Int64Counter (unit `{request}`), emitted by atelet. Each `EnsureImage` lookup counts one datapoint: * `ate.imagecache.outcome` (new key): `hit` when the node holds a complete image record — each layer directory that the record names is present — and `miss` when the lookup must pull. A failed lookup is neither, so it gets its own outcome, as no_free_worker does on the scheduler: `error` is a failed lookup whatever the cause (the registry, the ref, or the node), `cancelled` and `timeout` are the caller giving up, which ate.router.outcome also reports as outcomes. The hit ratio is thus hit / (hit + miss), with failures and abandoned lookups out of the denominator. * `error.type`: set only on the `error` outcome, and only from the registry's own HTTP status for its rejection, which is the one domain status this path has, as the gRPC code is for ateapi. transport.Error reports whatever the remote returned, so an allow-list bounds the label: 401, 403, 404, 429 and the 5xx set, each with its own operator action. Each other status, and each failure that carries none, reports `_OTHER`, the OTel registry's fallback. The atelet log names what broke, and ate.actor.restore.duration attributes it to the oci_unpack phase with the template beside it. The counter carries no identity labels. The layer pool is node state that each actor on the node shares, whatever its template or its sandbox class, so a class label would split one cache state into per-class series and give the pull to the class that asked first. The node and the pod already arrive as resource attributes, which is what per-node analysis of the hit ratio needs. `EnsureImage` records the datapoint in a deferred call, on a context that survives cancellation: a cancelled pull was started and paid for, and to drop it would hide a node that cancels each Resume. The store reports only when the caller gives it a meter (`WithMeter`), so the validation tool and the tests need no metrics pipeline. Tests: `internal/imagecache/metrics_test.go` collects the counter through a ManualReader-backed provider (miss then two hits, the outcome and the error.type of each failure kind, an unlisted status folding into `_OTHER`, and a store without a meter). `internal/ateattr` pins the wire spelling of the new key and its values. Docs: the metric table and the label notes in `docs/observability.md`.
Jeff Luo (JeffLuoo)
force-pushed
the
imagecache-requests-metric
branch
from
August 11, 2026 17:59
985efe6 to
e1b8a52
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the
ate.imagecache.requestscounter.ate.imagecache.outcome(new key ininternal/ateattr)hitmisserrorerror.typecancelled,timeoutA failed lookup is neither a hit nor a miss, so it gets its own outcome, as
no_free_workerdoes onate.scheduler.outcome.cancelledandtimeoutareoutcomes for the same reason they are on
ate.router.outcome. The hit ratio istherefore
hit / (hit + miss), with failures and abandoned lookups out of thedenominator.
error.type— set only on theerroroutcome.404,401,429, ..._OTHERFixes #831