feat: add execution context attributes to telemetry spans#288
Open
feat: add execution context attributes to telemetry spans#288
Conversation
pkosiec
force-pushed
the
pkosiec/obo-trace
branch
from
c5ada0c to
3914b27
Compare
Add `execution.context` and `caller.id` span attributes to the telemetry interceptor, allowing traces to distinguish OBO (user) from service principal code paths. Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
…Stream The TelemetryInterceptor spans were orphaned because OTel lost the parent HTTP span context when crossing into the async generator. Capture context.active() before the generator and restore it with context.with() inside, so plugin.execute spans appear as children of the HTTP request trace. Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
When asUser() is called in dev mode without x-forwarded-access-token, the telemetry span now includes execution.obo_dev_fallback: true to distinguish intended OBO calls from regular service principal calls. Uses OTel context key + thin proxy pattern to carry the flag without mutable state — scoped automatically per execution and concurrent-safe. Also documents telemetry span attributes in execution-context.md. Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
…reservation Add tests for previously uncovered behaviors: - asUser() dev fallback Proxy wraps methods correctly and sets isDevOboFallback context - EXCLUDED_FROM_PROXY methods bypass OBO fallback wrapping - executeStream preserves parent OTel context across async generator boundary - isDevOboFallback() returns false outside proxy context Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
The caller.id attribute was using context.userKey which is a cache key, not always the real user ID. The analytics plugin passes "global" for SP queries, so traces showed caller.id: "global" instead of the actual service principal ID. Now uses getCurrentUserId() which always returns the real identity. Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
…ement Add telemetry span attributes table to execution-context.md and note that execute()/executeStream() is required for automatic instrumentation. Update custom-plugins.md to link telemetry attributes from the execution interceptors bullet. Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
Add db.user attribute to lakebase.query telemetry spans so traces show which PostgreSQL role executed the query. Also add a comment clarifying that the arrow-result route intentionally bypasses the interceptor chain (it's a data download, not a query execution). Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
pkosiec
force-pushed
the
pkosiec/obo-trace
branch
2 times, most recently
from
bef4d7a to
97aba49
Compare
If getCurrentUserId() or isInUserContext() threw before this change, span.end() was never called because the calls were outside the try/finally block. Now all setAttribute calls are inside the try block, so the finally block guarantees span cleanup on any error. Signed-off-by: Pawel Kosiec <pawel.kosiec@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
execution.context,caller.id, andexecution.obo_dev_fallbackspan attributes to the telemetry interceptor for OBO observabilityplugin.executespans by preserving OTel context across the async generator boundary inexecuteStream()db.userattribute to lakebase query spansChanges
Telemetry span attributes (
plugin.executespan)execution.context"user"|"service"caller.idstringexecution.obo_dev_fallbackbooleantruewhen an OBO call falls back to SP in dev mode (nox-forwarded-access-token)OTel context propagation fix
executeStream()runs the interceptor chain inside an async generator — OTel lost the parent HTTP span context at this boundary. Spans were created but orphaned in separate traces. Fix: captureotelContext.active()before the generator, restore withotelContext.with()inside.OBO dev mode fallback
In dev mode without
x-forwarded-access-token,asUser()returns a thin proxy that sets an OTel context key (DEV_OBO_FALLBACK_KEY). The telemetry interceptor reads this to setexecution.obo_dev_fallback: true. No mutable state — scoped automatically per execution via OTel context.caller.idfixPreviously used
context.userKey(a cache key — analytics passes"global"for SP queries). Now usesgetCurrentUserId()which always returns the real identity.Lakebase
db.userAdds
db.userattribute tolakebase.queryspans showing the PostgreSQL role used for the connection.Arrow-result clarification
Added comment clarifying the arrow-result route intentionally bypasses the interceptor chain (data download, not query execution).
Test plan
Screenshots
Service Principal
User (for deployed app - when we have the forwarded token)
Local for OBO
Lakebase - DB user
Plugin OBO coverage verification
All built-in plugins that use
asUser()go throughexecute()/executeStream():executeStream()executeStream()execute()execute()execute()db.userattribute added tolakebase.queryspans showing the PostgreSQL role.execution.context/caller.idwill be added when OBO (per-user pools) is implemented.Not covered (by design):
GET /arrow-result/:jobId) — data download endpoint, always runs as SP. Clarified with comment.