feat: add telemetry intelligence signals for debugging and improvements

[anandgupta42]

What does this PR do?

Adds 7 new PII-safe telemetry events to improve product debugging and understanding of how users interact with the CLI:

1. Task Outcome Signal — behavioral signal (accepted/error/abandoned/cancelled) derived from session outcome
2. Task Intent Classification — regex/keyword classifier for user intent (debug_dbt, write_sql, optimize_query, etc.)
3. Tool Chain Effectiveness — ordered tool sequence with error/recovery tracking per session
4. Error Pattern Fingerprint — SHA256-hashed error messages with recovery tool linking
5. SQL Structure Fingerprint — AST shape via altimate-core (table count, aggregation, subqueries, node count)
6. dbt Project Fingerprint — materialization distribution, snapshot/seed counts added to environment_census
7. Schema Complexity Signal — bucketed table/column/schema counts from warehouse introspection

All signals computed locally from data already in memory — zero extra API calls, zero extra warehouse queries, <5ms overhead per session.

Type of change

• New feature (non-breaking change that adds functionality)

Issue for this PR

Closes #563

How did you verify your code works?

• 123 new telemetry-specific unit tests (all pass)
• 5,400 total test suite (0 failures)
• Adversarial testing: PII safety verification, edge cases (empty input, unicode, SQL injection, long strings)
• E2E session simulation covering all 7 signals in realistic order
• Privacy review: no PII, no table/column names, no SQL content, no user prompts, no file paths
• 6-model code review (Claude, GPT 5.2 Codex, Gemini 3.1 Pro, Kimi K2.5, MiniMax M2.5, GLM-5)
• TypeScript typecheck passes (0 errors)

Checklist

• My code follows the project's coding standards
• I have performed a self-review of my code
• New and existing tests pass locally
• I have added tests that prove my fix/feature works

Summary by CodeRabbit

• New Features

  • Enhanced session analytics to track task intent classification, outcome quality, and tool-chain effectiveness.
  • Improved error tracking and fingerprinting for better insights into execution failures and recovery patterns.
  • Added structural analysis for SQL queries and database schemas to better understand workload characteristics.

• Documentation

  • Updated telemetry reference with new event types and enhanced metrics for environment and project analysis.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

_{Tip: disable this comment in your organization's Code Review settings.}

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR introduces seven new PII-safe telemetry events and helper functions to improve product intelligence: task intent classification, outcome quality signals, tool chain effectiveness, error fingerprinting, SQL structure analysis, schema complexity metrics, and expanded dbt fingerprinting. It integrates emission logic across session, tool, and schema flows with comprehensive test coverage.

Changes

Cohort / File(s)	Summary
Telemetry Core Infrastructure packages/opencode/src/altimate/telemetry/index.ts	Extended Telemetry.Event union with six new event types (task_outcome_signal, task_classified, tool_chain_outcome, error_fingerprint, sql_fingerprint, schema_complexity). Added three exported helper functions: hashError() for SHA256-based error grouping, classifyTaskIntent() for regex-based intent classification with confidence scoring, and deriveQualitySignal() for outcome-to-signal mapping. Enhanced environment_census event with optional dbt metrics (snapshot/seed/macro counts and materialization distribution).
Tool Telemetry Integration packages/opencode/src/altimate/tools/project-scan.ts, packages/opencode/src/altimate/tools/sql-classify.ts, packages/opencode/src/altimate/tools/sql-execute.ts, packages/opencode/src/altimate/native/schema/register.ts	Added SQL fingerprinting utility (computeSqlFingerprint()) capturing statement types, categories, table/function counts, and feature flags. Integrated SQL fingerprint emission in sql-execute after query formatting (fail-closed). Extended project-scan with dbt manifest metrics (snapshot/seed counts and materialization distribution). Added schema_complexity event emission in schema indexing with bucketed table/column/schema counts and average columns-per-table ratio.
Session-Level Instrumentation packages/opencode/src/session/prompt.ts	Added comprehensive session instrumentation: task intent classification at session start (step 1) emitting task_classified event with warehouse type. Introduced per-step tool tracking with error detection, fingerprinting, and recovery recording. At session completion, emits task_outcome_signal (quality + timing), tool_chain_outcome (serialized tool chain + error/recovery counts), and up to 20 unique error_fingerprint events with recovery status.
Test Coverage packages/opencode/test/altimate/telemetry-moat-signals.test.ts, packages/opencode/test/telemetry/telemetry.test.ts	Added extensive Bun integration test suite (telemetry-moat-signals.test.ts) intercepting Telemetry.track to validate signal outputs, SQL fingerprinting behavior (aggregation/subquery/window function detection, PII safety), dbt metrics bucketing, schema complexity, and end-to-end session simulation. Expanded unit tests in telemetry.test.ts covering deriveQualitySignal, classifyTaskIntent, and computeSqlFingerprint with edge cases and error resilience.
Documentation & Configuration docs/docs/reference/telemetry.md, packages/opencode/package.json, .github/meta/commit.txt	Updated telemetry reference documentation with descriptions of six new event types and expanded environment_census fields. Bumped @altimateai/altimate-core dependency from 0.2.5 to 0.2.6. Updated commit message metadata reflecting new telemetry event addition.

Sequence Diagram(s)

sequenceDiagram
    participant Session as Session<br/>(prompt.ts)
    participant Telemetry as Telemetry<br/>(index.ts)
    participant Tools as Tools<br/>(sql-execute,<br/>project-scan,<br/>schema.index)
    participant Core as Core<br/>(`@altimateai/`<br/>altimate-core)

    Session->>Session: Step 1: Extract user text
    activate Session
    Session->>Telemetry: classifyTaskIntent(user_text)
    activate Telemetry
    Telemetry->>Core: regex/keyword matching
    Telemetry-->>Session: {intent, confidence}
    deactivate Telemetry
    Session->>Telemetry: track(task_classified)
    deactivate Session

    Session->>Tools: Execute tool calls
    activate Tools
    Tools->>Core: sql parse/classify
    Core-->>Tools: fingerprint data
    Tools->>Telemetry: track(sql_fingerprint)
    Tools->>Telemetry: track(schema_complexity)
    Tools->>Telemetry: track(environment_census)
    deactivate Tools

    Session->>Session: Track errors & recovery
    activate Session
    Session->>Telemetry: hashError(masked_error)
    activate Telemetry
    Telemetry-->>Session: error_hash
    deactivate Telemetry
    deactivate Session

    Session->>Telemetry: Session ends
    activate Telemetry
    Telemetry->>Telemetry: deriveQualitySignal(outcome)
    Telemetry->>Telemetry: track(task_outcome_signal)
    Telemetry->>Telemetry: track(tool_chain_outcome)
    Telemetry->>Telemetry: track(error_fingerprint[])
    deactivate Telemetry

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• feat: [AI-5975] add sql_quality telemetry for issue prevention metrics #446: Modifies the same telemetry surface (packages/opencode/src/altimate/telemetry/index.ts) and tool emission paths to add new event types and helper functions.
• feat: add core_failure telemetry with PII-safe input signatures #245: Expands the telemetry event union and introduces new telemetry helpers alongside existing infrastructure.

Suggested reviewers

• suryaiyer95
• mdesmet
• kulvirgit

Poem

🐰 Hop along, dear data flows!
Seven signals bloom where telemetry grows,
Intent classified, quality cast,
Error fingerprints holding the past—
All PII-safe, locally computed fast! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'feat: add telemetry intelligence signals for debugging and improvements' accurately and concisely describes the main change: adding 7 new telemetry events for product debugging and understanding.
Description check	✅ Passed	The PR description comprehensively covers all required template sections: summary of 7 telemetry signals, type of change (new feature), linked issue #563, detailed verification approach including 123 tests and privacy review, and completed checklist items.
Linked Issues check	✅ Passed	The pull request fully implements all coding requirements from issue #563: 7 PII-safe telemetry events (task outcome, intent classification, tool chain, error fingerprinting, SQL structure, dbt project expansion, schema complexity) with comprehensive testing and privacy verification.
Out of Scope Changes check	✅ Passed	All changes are directly aligned with the linked issue objectives: telemetry event definitions, helper functions, integration into session/tool flow, documentation updates, dependency bump for parsing fixes, and comprehensive test coverage—no extraneous modifications detected.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/telemetry-quality-signal

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gitguardian]

⚠️ GitGuardian has uncovered 3 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
29326785	Triggered	Generic Password	85cfe70	packages/opencode/test/altimate/connections.test.ts	View secret
29327894	Triggered	Generic Database Assignment	85cfe70	packages/opencode/test/altimate/connections.test.ts	View secret
29327588	Triggered	Generic Password	85cfe70	packages/opencode/test/altimate/connections.test.ts	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}