Count how many people run the template, and see who uses the demo

[jerelvelarde]

The problem

The 9 October launch brought more than a thousand unique cloners and roughly 2,800 stars, and we still cannot say how many people ran the template. The CopilotKit runtime inside it does report usage, but it sends about one event in twenty and nothing in those events says which app sent them, so this template's runs blend into every other CopilotKit app. The best available number was an estimate: runtimes on the pre-release version this template pins, which also includes the hosted demo.

The hosted demo was not measured at all. Nothing in the app loads analytics, so every visit to opengenerativeui.copilotkit.ai went unrecorded, and the launch dashboard has carried a "not tracked" row for it since day one.

The approach

Two events, once per installation. The frontend server sends oss.runtime.setup_step_viewed (step ready) the first time the UI reaches the runtime, and oss.runtime.setup_activated when the first agent run finishes with an assistant answer and no run error. Both go to the CopilotKit telemetry ingest unsampled. They sit under the oss.runtime.* namespace the ingest already accepts, so no sink change is needed and they reach PostHog alongside the existing runtime events. A random installation id is stored in apps/app/.open-intelligent-ui/telemetry.json (git-ignored), so counting distinct installation ids answers "how many people ran it", and the second event answers "how many got an answer".

Activation reads the response, not the request. Counting sent prompts would also count people whose keys are missing or whose agent is down. For the first agent run only, the route tees the response stream and reads event types (TEXT_MESSAGE_CONTENT, RUN_ERROR, RUN_FINISHED); no message text is kept. Once an installation has activated, the route stops inspecting requests. The cost is real and worth naming: the first run per installation parses its request body a second time and streams through a tee.

The runtime's own telemetry is tagged too. The pinned runtime (1.55.2-next.1) has no telemetryProperties option, but its Next.js endpoint copies properties._copilotkit into its telemetry properties. Passing the template name, distribution, version and installation id there tags the sampled runtime events without upgrading CopilotKit.

Opt-out and failure behave like the runtime. DO_NOT_TRACK, COPILOTKIT_TELEMETRY_DISABLED or a sample rate of 0 send nothing and delete the stored id. Delivery runs after the response through after(), times out at 1.5 seconds, never throws into a request, and is retried on a later request if it fails. The telemetry object is created on first use, so builds never touch the filesystem. The template now reports usage by default; docs/telemetry.md lists every field and is linked from the README.

Hosted-demo page analytics only where we host it. posthog-js loads only when NEXT_PUBLIC_POSTHOG_KEY is set at build time, so clones do not load it. Session recording is off because visitors type provider API keys into this app. render.yaml sets OIUI_DISTRIBUTION=hosted_demo, which keeps the demo's own runs out of the self-hosted count, and declares the PostHog key as a Render secret.

What is not covered

• Someone who clones and builds but never opens the app is not counted; the first event needs a request to the runtime.
• Render's filesystem is ephemeral, so each demo deploy or restart creates a new installation id. These carry hosted_demo and are excluded from install counts.
• Runs before this change cannot be counted retroactively.
• The demo records page views and PostHog's autocapture only. Prompt-level product events would be a follow-up.
• Page analytics start only after NEXT_PUBLIC_POSTHOG_KEY is set on the Render frontend service and the service is rebuilt.
• In the live browser check the test tab was hidden, so PostHog deferred its first page view; an autocapture event confirmed delivery and the app tag.

Verification

• pnpm test: app 164 passed (140 on main plus 24 new), design system 15, MCP 4.
• pnpm lint: no errors; the one warning (no-page-custom-font) is already on main.
• pnpm build: passes, and creates no telemetry state.
• src/lib/__tests__/launch-telemetry.test.ts holds down once-per-installation delivery, concurrent calls, persistence across restarts, retry after failure, every opt-out (including deleting stored state), CPK_TELEMETRY_ID as identity, the closed metadata keys, an unwritable state path, and the stream reader across split chunks, text chunks, run errors and runs with no answer.
• src/app/__tests__/posthog-analytics.test.tsx holds down that nothing loads without a key, that a key starts PostHog with session recording off and the app tag, and that it never starts twice.
• Live, against a production build of this branch and a stand-in AG-UI agent: both install events reached PostHog project 26816 with distribution development, which is excluded from counts. With COPILOTKIT_TELEMETRY_SAMPLE_RATE=1, tagged instance_created, copilot_request_created and agent_execution_stream_started/ended events arrived too. A build with the PostHog key delivered an autocapture event tagged app=open-intelligent-ui. The live run caught a bug the unit tests missed: the runtime streams text chunks rather than bytes, which the first version of the stream reader rejected; a test now covers it.