Add modular Langfuse-backed AI observability#82
Open
oshorefueled wants to merge 6 commits intomainfrom
Open
Conversation
📝 WalkthroughWalkthroughThis PR introduces optional Langfuse-based OpenTelemetry observability integration to the codebase. It adds observability interfaces, factory implementations, CLI lifecycle management, provider decoration logic, environment validation, and comprehensive test coverage. Observability initialization is non-blocking with graceful fallback to no-op on failure. Changes
Sequence DiagramsequenceDiagram
participant CLI as CLI Main
participant ObsFactory as Observability Factory
participant Obs as AIObservability
participant Provider as VercelAIProvider
participant VSDKas Vercel AI SDK
participant LF as Langfuse/OTEL
CLI->>ObsFactory: createObservability(env, logger)
ObsFactory->>Obs: new LangfuseObservability(config)
CLI->>Obs: init()
Obs->>LF: NodeSDK.start()
LF-->>Obs: initialized
CLI->>Provider: createProvider(env, {observability})
Provider-->>CLI: provider instance
CLI->>Provider: runPromptStructured(...)
Provider->>Provider: getObservabilityOptions(context)
Provider->>Obs: decorateCall({operation, provider, model, evaluator, rule})
Obs-->>Provider: {experimental_telemetry: {...}}
Provider->>VSDKas: generateText(..., experimental_telemetry)
VSDKas->>LF: record call with telemetry
LF-->>VSDKas: response
VSDKas-->>Provider: result
Provider-->>CLI: evaluation complete
CLI->>Obs: shutdown()
Obs->>LF: NodeSDK.shutdown()
LF-->>Obs: shutdown complete
CLI->>CLI: process.exit(code)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~35 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
README.md (1)
143-146: Add an explicit sensitive-data caution in observability notes.Since prompts/outputs are recorded, add a short warning to avoid sending secrets/PII unless organizational policy allows it.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@README.md` around lines 143 - 146, Update the Observability notes block in README.md (the "Observability is non-blocking..." / "Prompts and outputs are recorded..." lines) to add a concise sensitive-data warning: a one-line caution advising users not to include secrets, credentials, or PII in prompts/outputs unless their organizational policy explicitly permits it, and flag that recorded data may be accessible by observability tooling; keep the tone brief and add it immediately after the existing recorded-data sentence.src/schemas/env-schemas.ts (1)
66-71: Extract observability backend literals into a shared constant/type.
'langfuse'is repeated in schema and refinement checks; centralizing it reduces drift across runtime/config modules.♻️ Suggested refactor
+export const OBSERVABILITY_BACKENDS = ['langfuse'] as const; +export type ObservabilityBackend = (typeof OBSERVABILITY_BACKENDS)[number]; + const OBSERVABILITY_ENV_SCHEMA = z.object({ - OBSERVABILITY_BACKEND: z.enum(['langfuse']).optional(), + OBSERVABILITY_BACKEND: z.enum(OBSERVABILITY_BACKENDS).optional(), LANGFUSE_PUBLIC_KEY: z.string().min(1).optional(), LANGFUSE_SECRET_KEY: z.string().min(1).optional(), LANGFUSE_BASE_URL: z.string().url().optional(), }); @@ - if (data.OBSERVABILITY_BACKEND === 'langfuse') { + if (data.OBSERVABILITY_BACKEND === OBSERVABILITY_BACKENDS[0]) {As per coding guidelines "Define shared domain constants, enums, or types for core runtime concepts; avoid magic strings in shared runtime code".
Also applies to: 98-114
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/schemas/env-schemas.ts` around lines 66 - 71, Define a shared constant/type for observability backends (e.g., OBSERVABILITY_BACKENDS or ObservabilityBackend) and replace the hard-coded literal 'langfuse' used in OBSERVABILITY_ENV_SCHEMA and any related runtime/refinement checks (references: OBSERVABILITY_ENV_SCHEMA and the refinements around lines ~98-114) with that constant/type so both the z.enum(...) and all runtime validations import and use the same source of truth; update the schema definition to reference the shared constant values and change the refinement/validation code to use the same exported enum/union to avoid duplicate magic strings.src/boundaries/env-parser.ts (1)
31-31: Derive the allowed provider list from a shared constant instead of hard-coding it here.This message is now another place that must be updated whenever provider support changes. Please source the allowed values from the same shared runtime constant/enum that drives validation so the error text cannot drift again.
As per coding guidelines, "Define shared domain constants, enums, or types for core runtime concepts; avoid magic strings in shared runtime code."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/boundaries/env-parser.ts` at line 31, The error message in env-parser.ts hardcodes provider names; replace the literal list with a derived string from the shared provider constant/enum used for validation (e.g., SUPPORTED_LLM_PROVIDERS or LLMProviderType) so the message cannot drift—import that constant/enum, join its values to produce the allowed list, and use providerType (as in the current return) for the received value in the final string.src/observability/factory.ts (1)
12-13: Use the repository error type here instead of throwing rawError.This is still a configuration validation failure, so surfacing it as a native exception makes observability init inconsistent with the rest of the boundary/error handling.
Suggested change
+import { ValidationError } from '../errors/index'; import type { Logger } from '../logging/logger'; import type { EnvConfig } from '../schemas/env-schemas'; import type { AIObservability } from './ai-observability'; import { LangfuseObservability } from './langfuse-observability'; import { NoopObservability } from './noop-observability'; @@ if (!env.LANGFUSE_PUBLIC_KEY || !env.LANGFUSE_SECRET_KEY) { - throw new Error('Langfuse observability requires LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY'); + throw new ValidationError('Langfuse observability requires LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY'); }As per coding guidelines, "Prefer the repository's custom error hierarchy over native
Error."🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/observability/factory.ts` around lines 12 - 13, Replace the raw throw new Error(...) in the observability init check with the repository's custom configuration/error class: import the project's config/validation error type (e.g., ConfigError or RepositoryError) from the central errors module and throw that instead, using the same message that references env.LANGFUSE_PUBLIC_KEY and env.LANGFUSE_SECRET_KEY so the validation failure is surfaced via the repo's error hierarchy; ensure the import and thrown type replace the current throw new Error call in the LANGFUSE key check.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/boundaries/env-parser.ts`:
- Around line 62-69: The Langfuse branch currently treats any issue whose path
starts with "LANGFUSE_" as a missing credential; update the logic in the
env-parser branch that checks envObj.OBSERVABILITY_BACKEND === 'langfuse' (the
block that builds langfuseFields and missingLangfuseFields) to only
collect/report missing variables for the two credential keys LANGFUSE_PUBLIC_KEY
and LANGFUSE_SECRET_KEY (e.g., filter issue.path to exactly those two names and
only include issues with a "missing" type), and do not swallow other LANGFUSE_*
validation errors so they will be handled by the generic invalid-value formatter
instead. Ensure the message still references the two credential names and that
other Langfuse validation failures are left to the existing generic error path.
---
Nitpick comments:
In `@README.md`:
- Around line 143-146: Update the Observability notes block in README.md (the
"Observability is non-blocking..." / "Prompts and outputs are recorded..."
lines) to add a concise sensitive-data warning: a one-line caution advising
users not to include secrets, credentials, or PII in prompts/outputs unless
their organizational policy explicitly permits it, and flag that recorded data
may be accessible by observability tooling; keep the tone brief and add it
immediately after the existing recorded-data sentence.
In `@src/boundaries/env-parser.ts`:
- Line 31: The error message in env-parser.ts hardcodes provider names; replace
the literal list with a derived string from the shared provider constant/enum
used for validation (e.g., SUPPORTED_LLM_PROVIDERS or LLMProviderType) so the
message cannot drift—import that constant/enum, join its values to produce the
allowed list, and use providerType (as in the current return) for the received
value in the final string.
In `@src/observability/factory.ts`:
- Around line 12-13: Replace the raw throw new Error(...) in the observability
init check with the repository's custom configuration/error class: import the
project's config/validation error type (e.g., ConfigError or RepositoryError)
from the central errors module and throw that instead, using the same message
that references env.LANGFUSE_PUBLIC_KEY and env.LANGFUSE_SECRET_KEY so the
validation failure is surfaced via the repo's error hierarchy; ensure the import
and thrown type replace the current throw new Error call in the LANGFUSE key
check.
In `@src/schemas/env-schemas.ts`:
- Around line 66-71: Define a shared constant/type for observability backends
(e.g., OBSERVABILITY_BACKENDS or ObservabilityBackend) and replace the
hard-coded literal 'langfuse' used in OBSERVABILITY_ENV_SCHEMA and any related
runtime/refinement checks (references: OBSERVABILITY_ENV_SCHEMA and the
refinements around lines ~98-114) with that constant/type so both the
z.enum(...) and all runtime validations import and use the same source of truth;
update the schema definition to reference the shared constant values and change
the refinement/validation code to use the same exported enum/union to avoid
duplicate magic strings.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bf639973-83c8-40ff-bcf7-d1ddf9ebcd99
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (21)
.env.exampleCONFIGURATION.mdREADME.mdpackage.jsonsrc/boundaries/env-parser.tssrc/cli/commands.tssrc/observability/ai-observability.tssrc/observability/factory.tssrc/observability/langfuse-observability.tssrc/observability/noop-observability.tssrc/providers/provider-factory.tssrc/providers/vercel-ai-provider.tssrc/schemas/env-schemas.tstests/env-parser.test.tstests/main-command-observability.test.tstests/observability/factory.test.tstests/observability/langfuse-observability.test.tstests/observability/noop-observability.test.tstests/provider-factory.test.tstests/providers/vercel-ai-provider-agent-loop.test.tstests/vercel-ai-provider.test.ts
Comment on lines
+62
to
+69
| if (envObj.OBSERVABILITY_BACKEND === 'langfuse') { | ||
| const langfuseFields = issues | ||
| .filter((issue) => issue.path.length > 0 && String(issue.path[0]).startsWith('LANGFUSE_')) | ||
| .map((issue) => issue.path.join('.')); | ||
| const missingLangfuseFields = [...new Set(langfuseFields)]; | ||
|
|
||
| if (missingLangfuseFields.length > 0) { | ||
| return `Missing required Langfuse observability environment variables: ${missingLangfuseFields.join(', ')}. When using OBSERVABILITY_BACKEND=langfuse, ensure LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set.`; |
There was a problem hiding this comment.
This Langfuse branch mislabels non-credential validation failures as “missing required variables.”
The filter matches any LANGFUSE_* issue path, so an invalid LANGFUSE_BASE_URL will also hit this branch and produce the wrong message. Restrict this path to the two custom missing-credential issues and let other Langfuse validation errors fall through to the generic invalid-value formatter.
Suggested narrowing
if (envObj.OBSERVABILITY_BACKEND === 'langfuse') {
- const langfuseFields = issues
- .filter((issue) => issue.path.length > 0 && String(issue.path[0]).startsWith('LANGFUSE_'))
- .map((issue) => issue.path.join('.'));
- const missingLangfuseFields = [...new Set(langfuseFields)];
+ const missingLangfuseFields = [...new Set(
+ issues
+ .filter((issue) =>
+ issue.code === 'custom' &&
+ (issue.path[0] === 'LANGFUSE_PUBLIC_KEY' || issue.path[0] === 'LANGFUSE_SECRET_KEY')
+ )
+ .map((issue) => issue.path.join('.'))
+ )];
if (missingLangfuseFields.length > 0) {
return `Missing required Langfuse observability environment variables: ${missingLangfuseFields.join(', ')}. When using OBSERVABILITY_BACKEND=langfuse, ensure LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set.`;
}
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if (envObj.OBSERVABILITY_BACKEND === 'langfuse') { | |
| const langfuseFields = issues | |
| .filter((issue) => issue.path.length > 0 && String(issue.path[0]).startsWith('LANGFUSE_')) | |
| .map((issue) => issue.path.join('.')); | |
| const missingLangfuseFields = [...new Set(langfuseFields)]; | |
| if (missingLangfuseFields.length > 0) { | |
| return `Missing required Langfuse observability environment variables: ${missingLangfuseFields.join(', ')}. When using OBSERVABILITY_BACKEND=langfuse, ensure LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set.`; | |
| if (envObj.OBSERVABILITY_BACKEND === 'langfuse') { | |
| const missingLangfuseFields = [...new Set( | |
| issues | |
| .filter((issue) => | |
| issue.code === 'custom' && | |
| (issue.path[0] === 'LANGFUSE_PUBLIC_KEY' || issue.path[0] === 'LANGFUSE_SECRET_KEY') | |
| ) | |
| .map((issue) => issue.path.join('.')) | |
| )]; | |
| if (missingLangfuseFields.length > 0) { | |
| return `Missing required Langfuse observability environment variables: ${missingLangfuseFields.join(', ')}. When using OBSERVABILITY_BACKEND=langfuse, ensure LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set.`; | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/boundaries/env-parser.ts` around lines 62 - 69, The Langfuse branch
currently treats any issue whose path starts with "LANGFUSE_" as a missing
credential; update the logic in the env-parser branch that checks
envObj.OBSERVABILITY_BACKEND === 'langfuse' (the block that builds
langfuseFields and missingLangfuseFields) to only collect/report missing
variables for the two credential keys LANGFUSE_PUBLIC_KEY and
LANGFUSE_SECRET_KEY (e.g., filter issue.path to exactly those two names and only
include issues with a "missing" type), and do not swallow other LANGFUSE_*
validation errors so they will be handled by the generic invalid-value formatter
instead. Ensure the message still references the two credential names and that
other Langfuse validation failures are left to the existing generic error path.
- create the runtime logger only for verbose runs or active telemetry - pass provider and model names explicitly into Vercel AI calls instead of reflecting on SDK model objects - centralize Langfuse backend constants and narrow missing-credential reporting to the required keys - document recorded-data caution and extend tests around logger and telemetry metadata behavior
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title
Add modular Langfuse-backed AI observabilitySummary
This adds a modular AI observability layer to VectorLint and wires it into the existing Vercel AI SDK execution paths. Langfuse is supported through in-process OpenTelemetry setup, observability failures stay non-blocking, and the repo now documents the new config plus the Node 20.6+ runtime requirement introduced by the Langfuse/OTEL dependency path.
Why
VectorLint had no structured way to observe AI executions, and wiring Langfuse directly into provider code would have coupled the core runtime to one backend. This adds a narrow observability seam so Langfuse can be enabled now without locking the architecture to a single platform.
Scope
AIObservabilitycontract with noop and Langfuse-backed implementations20.6+PerplexitySearchProviderBehavior Impact
yes- users can enable Langfuse observability withOBSERVABILITY_BACKEND=langfuseand the corresponding credentialsyes- the declared runtime support moves from Node 18+ to Node20.6+VercelAIProvidercalls and attempts a shutdown flush before process exitRisk and Mitigations
mediumvitest runOBSERVABILITY_BACKENDAPI / Contract / Schema Changes
OBSERVABILITY_BACKEND=langfuseLANGFUSE_PUBLIC_KEYLANGFUSE_SECRET_KEYLANGFUSE_BASE_URLAIExecutionContextAIObservabilitypackage.jsonengines updated to>=20.6.0Follow-ups
PerplexitySearchProvidershould also participate in the observability layersrc/cli/commands.tsto remove mutable exit bookkeepingKnown Tradeoffs
VercelAIProvider