Skip to content

feat(telemetry): track Argent Lens design-review funnel#478

Merged
latekvo merged 7 commits into
mainfrom
feat/lens-telemetry-enrichment
Jul 13, 2026
Merged

feat(telemetry): track Argent Lens design-review funnel#478
latekvo merged 7 commits into
mainfrom
feat/lens-telemetry-enrichment

Conversation

@hubgan

@hubgan hubgan commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

What

Adds three privacy-safe telemetry events for the Argent Lens design-review flow, capturing the human-decision funnel that the generic tool:* path can't see (submission is an HTTP POST, and the await_user_selection tool is hidden in argent lens CLI sessions):

  • lens:preview_opened — a human loaded the preview UI (GET /preview/), deduped per round.
  • lens:round_completed — the human submitted their picks (once per round, never on a resubmit).
  • lens:round_abandoned — a staged round was discarded before submit (window closed, argent lens exited mid-review, or the round superseded).

Privacy

Payloads carry only aggregate counts, booleans, durations, and the device platform enum — never element names, comment text, variant code, file paths, or raw device ids. The store emits privacy-safe aggregate structs that are decoupled from the telemetry package; structural assignability to the event props is enforced at the track() call site (drift is a compile error).

Event fields

  • preview_opened: round, element_count, variant_count, is_cli_session, platform
  • round_completed: round, element_count, variant_count, annotation_count, element_comment_count, skipped_comment_count, has_global_comment, is_cli_session, had_parked_await, round_duration_ms, platform
  • round_abandoned: round, element_count, variant_count, had_parked_await, is_cli_session, platform

Testing

  • Unit tests for all three events: per-round dedup, exactly-once completion (no resubmit re-emit), abandoned vs happy-path roll, CLI up-front open, and content-agnostic privacy assertions.
  • Verified end-to-end against a live tool-server: all three events fire with correct aggregate values and zero content leakage through the real sanitize pipeline.

hubgan added 2 commits July 7, 2026 13:08
Add three privacy-safe telemetry events for the Argent Lens flow that the
generic tool:* path can't capture (submission is an HTTP POST, and the
await tool is hidden in CLI sessions):

- lens:preview_opened  — a human loaded the preview UI (deduped per round)
- lens:round_completed — the human submitted their picks (once per round)
- lens:round_abandoned — a staged round was discarded before submit

Payloads carry only aggregate counts, booleans, durations, and the device
platform enum — never element names, comment text, variant code, paths, or
raw device ids. The store emits privacy-safe aggregate structs decoupled
from the telemetry package; structural assignability to the event props is
enforced at the track() call site.
- add a preview_opened test asserting per-round re-emit (not fire-once),
  guarding the round != lastOpenedRound dedup against regression
- add a round_completed test with an asymmetric comment case so
  element_comment_count (2) and skipped_comment_count (1) cannot be swapped
- de-flake the "emits once" preview_opened test: the store device singleton
  survives reset(), so a platform: undefined toEqual was order-dependent
  (failed under --sequence.shuffle); assert the controlled fields via
  toMatchObject plus an exact key-set guard instead
@hubgan hubgan requested a review from latekvo July 7, 2026 12:08

@latekvo latekvo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed end-to-end. Built the monorepo (tsc --build clean), ran the telemetry and tool-server suites plus both typecheck:tests gates (green - the one full-suite await-ui-element failure is a pre-existing timing flake that reproduces identically on unmodified main and is untouched here), and drove the real store -> real sanitize() pipeline with adversarial secret-laden inputs.

The privacy contract holds: every field is an aggregate count/boolean/duration or the coarse platform enum (classifyDevice never echoes a raw device id, even for a malformed one), with sanitize() as the enforced backstop. roundCompleted/roundAbandoned are exactly-once and mutually exclusive, the aggregate counts match their doc comments (including the asymmetric element_comment_count vs skipped_comment_count split), the compile-time drift guard between the store stats and the telemetry props genuinely fails on a mismatch, listener registration/removal is symmetric, and emission is throw-safe.

One item worth resolving is left inline: lens:preview_opened undercounts rounds in multi-round argent lens CLI sessions, which skews the open-to-complete funnel these events exist to measure. Approving, since the events are individually correct and privacy-safe and the rest of the change is thoroughly verified.

Comment thread packages/tool-server/src/preview.ts Outdated
…indow

In an `argent lens` CLI session the preview window opens once and is reused for
the whole session: the UI swaps rounds client-side off the /preview/variants
poll without ever reloading `/`, and await_user_selection is hidden so the
window is never re-foregrounded. `lens:preview_opened` was emitted only on
`GET /`, so it fired once per session while lens:round_completed/round_abandoned
fire once per round — the open-to-decision funnel showed more decisions than
opens for any 2+-round session.

Emit from the /variants poll too (the request only an open preview window makes
— the agent mutates the store in-process and never speaks HTTP), sharing the
lastOpenedRound dedup via a trackPreviewOpenedOnce helper so a round is counted
exactly once across both surfaces. The MCP path is unchanged: it respawns the
window per round, so `GET /` still fires first and the same-round poll no-ops.
@hubgan hubgan marked this pull request as ready for review July 7, 2026 13:32
Extend the Argent Lens funnel with two signals the existing events can't
answer:

- lens:round_completed gains `inspector_used` and `offscreen_revealed`
  booleans — whether the human opened the "Add comment" inspector or
  clicked "Show them" (reveal off-screen choices) during the round. The
  preview UI tracks sticky per-round flags (reset on a new round / after a
  submit) and rides them on the selection POST; the store forwards them
  into the round-completed aggregate. annotation_count only counts saved
  comments, so it cannot measure inspector adoption on its own.

- lens:cli_session_started fires once per `argent lens` invocation (the
  CLI session-begin), carrying agent_choice_count. The generic tool:* path
  counts per tool call and lens:preview_opened counts per round, so neither
  can answer how many times the tool was run or the unique-user population.
  Emitted on every begin (not only a clean transition) so an invocation
  that starts while a killed prior session is still marked active is not
  under-counted.

All payloads stay privacy-safe aggregates (counts/booleans) — never
element names, comment text, agent names, or device ids.
@hubgan hubgan requested a review from latekvo July 8, 2026 11:24
Comment thread packages/tool-server/src/utils/variant-proposals.ts Outdated
Comment thread packages/tool-server/src/utils/variant-proposals.ts Outdated
Comment thread packages/tool-server/src/utils/variant-proposals.ts
Comment thread packages/tool-server/src/utils/variant-proposals.ts Outdated
Comment thread packages/tool-server/src/utils/variant-proposals.ts Outdated
Comment thread packages/tool-server/src/preview.ts Outdated
Comment thread packages/telemetry/src/events.ts Outdated
Comment thread packages/telemetry/src/events.ts Outdated
Comment thread packages/telemetry/src/events.ts
Comment thread packages/tool-server/test/variant-proposals.test.ts
latekvo added 3 commits July 13, 2026 16:17
Address the review on the Lens design-review telemetry:

- preview_opened: drive it from an explicit, visibility-gated client
  signal (POST /preview/opened) instead of inferring it from `GET /`
  (undercounts reused CLI windows) or the /variants poll (a backgrounded
  tab overcounts; counts froze at the first poll tick). Counts/platform
  are read server-side; platform is omitted for a zero-proposal open so a
  stale device can't attribute a CLI up-front open.

- TV attribution: refine `ios`/`android` to `tvos`/`android-tv` for lens
  events via a shared `refineTvPlatform` (factored out of http.ts), so a
  Lens session on a TV target is attributed like its tool:* events.

- stale submit: carry the round the UI built a submit against and reject
  it when the round has rolled, so a late duplicate click can't mint a
  phantom round_completed for a round nobody reviewed.

- dead CLI session: sweep leftover round state on every `argent lens`
  begin (not only a clean transition), so a killed prior session's staged
  round is counted as abandoned and can't bleed into the new invocation.

- shutdown: flush a still-staged round as abandoned ("server died
  mid-review"), and keep the store-event relays attached until after the
  HTTP server closes so a submit during drain is still relayed.

- drift guard: add a bidirectional compile-time key-set check between the
  store stat structs and the telemetry props (the relay passes variables,
  so track()'s excess-property check alone missed added fields).

- docs: correct the cliSessionStarted docblock, the reset()/drift-guard
  comments, and the agent_choice_count 0/1 note; add a design-review
  category to the privacy notice.

- tests: relay-wiring runtime test (drop/cross-wire), CLI-exit and
  begin-sweep abandonment attribution, dead-session re-begin, stale-submit
  rejection (store + route), and the reworked preview_opened surface.
…nrichment

# Conflicts:
#	packages/tool-server/src/index.ts

@latekvo latekvo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed after the fixes in 379f92b (+ a main merge to resolve a shutdown conflict). All fourteen inline findings are addressed and every one is verified — not just believed:

  • preview_opened is now a single, visibility-gated client signal (POST /preview/opened): one open per round in a reused CLI window, no background-tab overcount, consistent count sampling across MCP/CLI, and platform omitted for a zero-proposal open so a surviving device can't attribute a CLI up-front open. GET / and the /variants poll no longer emit.
  • TV attribution goes through a shared refineTvPlatform (factored out of http.ts), so a Lens session on a TV target reports tvos/android-tv like its tool:* events.
  • Phantom completion is closed by a round token the store rejects when the round has rolled (route forwards it, UI sends it, usage flags reset on both paths).
  • Dead CLI session: leftover round state is swept on every begin, so a killed predecessor's staged round is counted abandoned and can't bleed forward.
  • Shutdown: relays stay attached until after server.close(), and a still-staged round is flushed as an abandonment before the drain.
  • Drift guard is now bidirectional (compile-time key-set check); the docblocks (cliSessionStarted, drift, agent_choice_count 0/1) and the reset() abandonment comment are corrected; the privacy notice gains a design-review category.

Verification: tsc --build clean; telemetry (259) and full tool-server (2201) suites green; both typecheck:tests gates, ESLint, and Prettier pass; the drift guard proven to bite via a probe field; and a live end-to-end run against the real preview router + store + relay + sanitize() pipeline confirmed each event fires with correct aggregate values, the stale submit is rejected with no phantom completion, the CLI-exit abandonment is labeled is_cli_session:true, the up-front open omits the stale platform, and no element name / comment text leaks. CI is green on the head commit.

Approving.

@latekvo latekvo merged commit f002fc9 into main Jul 13, 2026
5 checks passed
@latekvo latekvo deleted the feat/lens-telemetry-enrichment branch July 13, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants