feat(telemetry): track Argent Lens design-review funnel#478
Conversation
Add three privacy-safe telemetry events for the Argent Lens flow that the generic tool:* path can't capture (submission is an HTTP POST, and the await tool is hidden in CLI sessions): - lens:preview_opened — a human loaded the preview UI (deduped per round) - lens:round_completed — the human submitted their picks (once per round) - lens:round_abandoned — a staged round was discarded before submit Payloads carry only aggregate counts, booleans, durations, and the device platform enum — never element names, comment text, variant code, paths, or raw device ids. The store emits privacy-safe aggregate structs decoupled from the telemetry package; structural assignability to the event props is enforced at the track() call site.
- add a preview_opened test asserting per-round re-emit (not fire-once), guarding the round != lastOpenedRound dedup against regression - add a round_completed test with an asymmetric comment case so element_comment_count (2) and skipped_comment_count (1) cannot be swapped - de-flake the "emits once" preview_opened test: the store device singleton survives reset(), so a platform: undefined toEqual was order-dependent (failed under --sequence.shuffle); assert the controlled fields via toMatchObject plus an exact key-set guard instead
latekvo
left a comment
There was a problem hiding this comment.
Reviewed end-to-end. Built the monorepo (tsc --build clean), ran the telemetry and tool-server suites plus both typecheck:tests gates (green - the one full-suite await-ui-element failure is a pre-existing timing flake that reproduces identically on unmodified main and is untouched here), and drove the real store -> real sanitize() pipeline with adversarial secret-laden inputs.
The privacy contract holds: every field is an aggregate count/boolean/duration or the coarse platform enum (classifyDevice never echoes a raw device id, even for a malformed one), with sanitize() as the enforced backstop. roundCompleted/roundAbandoned are exactly-once and mutually exclusive, the aggregate counts match their doc comments (including the asymmetric element_comment_count vs skipped_comment_count split), the compile-time drift guard between the store stats and the telemetry props genuinely fails on a mismatch, listener registration/removal is symmetric, and emission is throw-safe.
One item worth resolving is left inline: lens:preview_opened undercounts rounds in multi-round argent lens CLI sessions, which skews the open-to-complete funnel these events exist to measure. Approving, since the events are individually correct and privacy-safe and the rest of the change is thoroughly verified.
…indow In an `argent lens` CLI session the preview window opens once and is reused for the whole session: the UI swaps rounds client-side off the /preview/variants poll without ever reloading `/`, and await_user_selection is hidden so the window is never re-foregrounded. `lens:preview_opened` was emitted only on `GET /`, so it fired once per session while lens:round_completed/round_abandoned fire once per round — the open-to-decision funnel showed more decisions than opens for any 2+-round session. Emit from the /variants poll too (the request only an open preview window makes — the agent mutates the store in-process and never speaks HTTP), sharing the lastOpenedRound dedup via a trackPreviewOpenedOnce helper so a round is counted exactly once across both surfaces. The MCP path is unchanged: it respawns the window per round, so `GET /` still fires first and the same-round poll no-ops.
Extend the Argent Lens funnel with two signals the existing events can't answer: - lens:round_completed gains `inspector_used` and `offscreen_revealed` booleans — whether the human opened the "Add comment" inspector or clicked "Show them" (reveal off-screen choices) during the round. The preview UI tracks sticky per-round flags (reset on a new round / after a submit) and rides them on the selection POST; the store forwards them into the round-completed aggregate. annotation_count only counts saved comments, so it cannot measure inspector adoption on its own. - lens:cli_session_started fires once per `argent lens` invocation (the CLI session-begin), carrying agent_choice_count. The generic tool:* path counts per tool call and lens:preview_opened counts per round, so neither can answer how many times the tool was run or the unique-user population. Emitted on every begin (not only a clean transition) so an invocation that starts while a killed prior session is still marked active is not under-counted. All payloads stay privacy-safe aggregates (counts/booleans) — never element names, comment text, agent names, or device ids.
Address the review on the Lens design-review telemetry:
- preview_opened: drive it from an explicit, visibility-gated client
signal (POST /preview/opened) instead of inferring it from `GET /`
(undercounts reused CLI windows) or the /variants poll (a backgrounded
tab overcounts; counts froze at the first poll tick). Counts/platform
are read server-side; platform is omitted for a zero-proposal open so a
stale device can't attribute a CLI up-front open.
- TV attribution: refine `ios`/`android` to `tvos`/`android-tv` for lens
events via a shared `refineTvPlatform` (factored out of http.ts), so a
Lens session on a TV target is attributed like its tool:* events.
- stale submit: carry the round the UI built a submit against and reject
it when the round has rolled, so a late duplicate click can't mint a
phantom round_completed for a round nobody reviewed.
- dead CLI session: sweep leftover round state on every `argent lens`
begin (not only a clean transition), so a killed prior session's staged
round is counted as abandoned and can't bleed into the new invocation.
- shutdown: flush a still-staged round as abandoned ("server died
mid-review"), and keep the store-event relays attached until after the
HTTP server closes so a submit during drain is still relayed.
- drift guard: add a bidirectional compile-time key-set check between the
store stat structs and the telemetry props (the relay passes variables,
so track()'s excess-property check alone missed added fields).
- docs: correct the cliSessionStarted docblock, the reset()/drift-guard
comments, and the agent_choice_count 0/1 note; add a design-review
category to the privacy notice.
- tests: relay-wiring runtime test (drop/cross-wire), CLI-exit and
begin-sweep abandonment attribution, dead-session re-begin, stale-submit
rejection (store + route), and the reworked preview_opened surface.
…nrichment # Conflicts: # packages/tool-server/src/index.ts
latekvo
left a comment
There was a problem hiding this comment.
Re-reviewed after the fixes in 379f92b (+ a main merge to resolve a shutdown conflict). All fourteen inline findings are addressed and every one is verified — not just believed:
- preview_opened is now a single, visibility-gated client signal (
POST /preview/opened): one open per round in a reused CLI window, no background-tab overcount, consistent count sampling across MCP/CLI, andplatformomitted for a zero-proposal open so a surviving device can't attribute a CLI up-front open.GET /and the/variantspoll no longer emit. - TV attribution goes through a shared
refineTvPlatform(factored out of http.ts), so a Lens session on a TV target reportstvos/android-tvlike itstool:*events. - Phantom completion is closed by a round token the store rejects when the round has rolled (route forwards it, UI sends it, usage flags reset on both paths).
- Dead CLI session: leftover round state is swept on every begin, so a killed predecessor's staged round is counted abandoned and can't bleed forward.
- Shutdown: relays stay attached until after
server.close(), and a still-staged round is flushed as an abandonment before the drain. - Drift guard is now bidirectional (compile-time key-set check); the docblocks (cliSessionStarted, drift, agent_choice_count 0/1) and the reset() abandonment comment are corrected; the privacy notice gains a design-review category.
Verification: tsc --build clean; telemetry (259) and full tool-server (2201) suites green; both typecheck:tests gates, ESLint, and Prettier pass; the drift guard proven to bite via a probe field; and a live end-to-end run against the real preview router + store + relay + sanitize() pipeline confirmed each event fires with correct aggregate values, the stale submit is rejected with no phantom completion, the CLI-exit abandonment is labeled is_cli_session:true, the up-front open omits the stale platform, and no element name / comment text leaks. CI is green on the head commit.
Approving.
What
Adds three privacy-safe telemetry events for the Argent Lens design-review flow, capturing the human-decision funnel that the generic
tool:*path can't see (submission is an HTTP POST, and theawait_user_selectiontool is hidden inargent lensCLI sessions):lens:preview_opened— a human loaded the preview UI (GET /preview/), deduped per round.lens:round_completed— the human submitted their picks (once per round, never on a resubmit).lens:round_abandoned— a staged round was discarded before submit (window closed,argent lensexited mid-review, or the round superseded).Privacy
Payloads carry only aggregate counts, booleans, durations, and the device
platformenum — never element names, comment text, variant code, file paths, or raw device ids. The store emits privacy-safe aggregate structs that are decoupled from the telemetry package; structural assignability to the event props is enforced at thetrack()call site (drift is a compile error).Event fields
preview_opened:round, element_count, variant_count, is_cli_session, platformround_completed:round, element_count, variant_count, annotation_count, element_comment_count, skipped_comment_count, has_global_comment, is_cli_session, had_parked_await, round_duration_ms, platformround_abandoned:round, element_count, variant_count, had_parked_await, is_cli_session, platformTesting