feat(simulate): platform scenarios + LiveKit SIP transport + provider acceptance flows - #55
Open
azain-commits wants to merge 16 commits into
Open
Conversation
…iveKit SIP Prompt & simulation: - Add fi.simulate.simulation.voice_prompt with call-direction-aware, role-locked, personality/style-aware persona prompt policy adapted from platform. - Make LiveKit engine use the SDK prompt by default; treat simulator.instructions as a full override; drop the ad-hoc opener that could read scenario text aloud. - Retain SIP outbound path: sip_number/wait_until_answered/ringtone, PARTICIPANT_KIND_SIP subscription, typed sip_dial_failed and sip_inbound_no_participant failures. Platform scenario generation (opt-in, keyed): - Add fi.alk.studio._generate with PlatformScenarioRequest, GeneratedScenario, ScenarioGenerationError, ensure_platform_agent, generate_scenario, fetch_scenario. Idempotent stable-name Agent Definition create/reuse, scanned safe metadata upload, async Scenario poll + dataset-table pagination. - Upgrade fi.alk.studio._download with real dataset-table hydration (map_dataset_table_rows, fetch_dataset_rows, hydrate_platform_scenario) and a shared row parser; fail closed instead of fabricating rows. - Export new public API from fi.alk.studio. CLI & manifests: - Add `agent-learn scenario generate` command (local AgentDefinition or explicit platform IDs, JSON output). - Add scenario.platform block to fi.simulate.cli._build_scenario with local cache reuse; incompatible with source/dataset; runs generation off-thread. Runtime scaffolding & artifacts: - New fi.simulate.runtime, fi.simulate.artifacts, fi.simulate.environments, fi.simulate.evidence, fi.simulate.results supporting canonical local text, LiveKit, and cloud engines; refactor engines and recording accordingly. - Add examples/build_delivery_support_suite.py and .github/workflows/sdk-smoke.yml. Tests: - New tests/runtime/ suites (cli_smoke, livekit_engine, manifest_engine_dispatch, runtime_contracts, simulation_runner, delivery_support_suite). No new scenario-generation tests in this pass (tracked as follow-up debt).
…ades Follow-up to previous commit that only picked up the new files: - src/fi/alk/cli.py: add `agent-learn scenario generate` subcommand and wire it to studio.generate_scenario, including local AgentDefinition and explicit platform ID paths, JSON output writing, and structured refusal on failure. - src/fi/alk/studio/__init__.py: export the new generation public API (PlatformAgentReference, PlatformScenarioRequest, GeneratedScenario, ScenarioGenerationError, ensure_platform_agent, generate_scenario, fetch_scenario) alongside existing symbols. - src/fi/alk/studio/_download.py: real dataset-table pagination and hydration (map_dataset_table_rows, fetch_dataset_rows, hydrate_platform_scenario) plus a shared row parser reused by generate_scenario; pull_scenarios uses the actual platform contract and fails closed instead of fabricating rows. - src/fi/alk/simulate.py: expose scenario generation surface alongside existing simulate helpers. - src/fi/simulate/agent/definition.py: TelephonyTransport (webrtc / sip_outbound / sip_inbound) with E.164 validation and wait_until_answered semantics; AgentDefinition.transport field. - src/fi/simulate/cli.py: add scenario.platform block to _build_scenario with local cache reuse and off-thread execution; incompatible with scenario.source / scenario.dataset. - src/fi/simulate/manifest.py: ancillary manifest plumbing for the new scenario shape. - .gitignore/pyproject.toml/uv.lock: keep new package layout clean and pin dependency changes required by the runtime + livekit paths.
… Vertex switch
Land the missing pieces called out in the simulation SDK implementation plan and
the LiveKit Cloud provider work that follows from it:
- Protocols & media: endpoints/base.py, realtime/{media,events,session}, and
simulator/ define AgentEndpoint, RealtimeEndpoint, AudioFrame,
SimulatorPolicy, the canonical event vocabulary, and the media profile.
- Endpoint adapters: callable/http/websocket/livekit/retell + Vapi originator
helper in endpoints/vapi.py (POST /call, DELETE /call/{id}, env resolution).
- Evidence sources: providers/{vapi,retell} adapters plus caller_observed,
livekit_room, livekit_instrumentation, and otel evidence skeletons.
- Instrumentation: instrumentation/livekit/FutureAGIObserver skeleton.
- Result sink: results/futureagi.py serialises the plan's §11.2 route contract
locally when FUTURE_AGI_API_URL is unset.
- Matrix runner: simulation/matrix.py + matrix_cli.py drive provider×channel
sweeps and now surface each leg's SimulationReport status so evaluator score
no longer masks failed runs.
- Vertex/Gemini backend: livekit_models.py adds Google LLM/STT/TTS factories
(Vertex when GOOGLE_APPLICATION_CREDENTIALS + GOOGLE_CLOUD_PROJECT are set,
Gemini API when GEMINI_API_KEY is set). Split speech creds from LLM Vertex
kwargs; default TTS voice moves to en-US-Chirp3-HD-Kore for streaming.
- Package plumbing: livekit extras include livekit-plugins-google; public
re-exports for endpoints/realtime/simulator/results/evidence added.
- Manifest tests exercise the SIP transport + Vapi/Retell evidence contracts.
… failures Previously the LiveKit engine could not run against a real LiveKit Cloud project: outbound dial hit a spurious ServerError on premature RoomService CreateRoom, inbound routed the SIP participant into a randomly-suffixed room that the local simulator was not in, the AgentSession subscribed to the simulator's own participant instead of the SIP caller, cleanup deleted the room before the dispatch rule, and the failure surface collapsed every setup error into a single opaque code. Fixes: - Outbound: skip CreateRoom for sip_outbound (LiveKit auto-creates on join); connect the simulator RTC first, start its session, then create the SIP participant. Split answer_timeout_seconds from connect_timeout so PSTN answer latency stops surfacing as generic connect timeouts. - Inbound: _ensure_sip_inbound_dispatch now creates a per-case SIPDispatchRuleDirect(room_name=<exact case room>) bound to LIVEKIT_INBOUND_TRUNK_ID and validates a reused named rule by trunk + direct-rule type + destination room. Adds sip_inbound_route_conflict. - Participant subscription: _TestRunnerAgent.start_session accepts participant_kinds/participant_identity; for SIP the engine passes [PARTICIPANT_KIND_SIP] and the caller identity so the AgentSession locks onto the actual caller, not the simulator itself. - Vapi call originator: new inbound_call_originator="vapi" on TelephonyTransport plus call_id_source="originator_response" on ProviderEvidenceConfig. Engine drives VapiCallOriginator after the direct dispatch and session are ready, passes the returned Vapi call ID into _collect_provider_evidence as the explicit hint, and cancels the Vapi call on cleanup. - Cleanup order: delete SDK-owned dispatch rule before the room; retain not-found tolerance. - Failures: typed sanitized codes for room create, SIP dial, SIP dispatch, and Vapi call start (livekit_room_create_failed, sip_dial_failed, sip_answer_timeout, sip_inbound_dispatch_failed, sip_inbound_no_participant, vapi_call_start_failed, vapi_call_start_timeout). _safe_provider_error_details keeps only exception type, provider code, HTTP status. - Grade natural target hang-ups: if the target disconnected and both roles spoke, treat the case as completed instead of target_disconnected. - AgentDefinition._check_transport validates origin-only ws/wss URL scheme and rejects SIP transports with room_mode="external". Focused tests cover new SIP behaviours, Vapi originator request shape, and sanitized error propagation.
…+ port Platform voice prompt
Split AgentDefinition into a target-agent shape (Vapi/Retell provider
config, non-secret assistant/agent id, api_key_env) and a new
LiveKitSimulatorRuntime that owns FutureAGI's LiveKit URL, room, and
key/secret env names. Legacy AgentDefinition.url/room_name/room_mode
stay as a compatibility boundary. Provider bridges and evidence
adapters now read credentials from the target's api_key_env and the
target's API base URL instead of hard-coded VAPI_/RETELL_ globals.
Ported the complete production voice persona prompt and execution
rules from the Platform (test_executor + ee.voice guides) into
fi.simulate.simulation.voice_prompt, and made
SimulatorAgentDefinition.instructions additive on top of the
scenario-derived customer prompt instead of replacing it.
Platform registration for a direct target now records the actual
provider ("vapi" or "retell") with its non-secret id and sets
scenario_generation_only, so no provider secrets or FutureAGI LiveKit
runtime credentials are uploaded during scenario generation. The
Platform description is the target agent's system prompt only.
Renamed the primary Platform credential env vars to FI_API_KEY,
FI_SECRET_KEY, and FI_BASE_URL. FUTURE_AGI_* and AGENT_LEARNING_*
remain as compatibility aliases.
CLI, manifest builder, examples, and tests updated to consume the
explicit target/runtime shape.
…ning Simulator engine now honors LiveKitSimulatorRuntime.room_name_verbatim so runs can join a pre-existing dispatch rule bound to a fixed room, which is what unblocks the 10-slot inbound-simulator DID pool used by the production LiveKit matrix. Multi-persona runs guard against verbatim reuse. trigger_livekit_outbound short-circuits room discovery when the matrix runner provides an override. Runner-side hardening: split simulator utterances are merged before turn-count evaluation, endCall refuses to fire before both speakers have participated (min_turn_messages becomes a validation floor rather than a stop trigger), and empty / silent / short conversations now surface as failures instead of passing. Matrix runner returns non-zero on behavioural evaluation failure. Provider fixes bundled: Cartesia STT/TTS support, Google endpoint + STT + LINEAR16 TTS corrections, authenticated Bearer retrieval for Vapi private recordings with signed-URL redirects, Vapi tool arg/result capture, corrected recording paths and manifest metadata, migration off deprecated LiveKit room/turn options, and unique worker names to avoid stale registrations.
Replaces the deferred submission stub with a working HTTP client that posts a finished simulation run to the FutureAGI platform's ALK ingestion endpoints: start a test execution, allocate call executions, upload each recording, then PATCH the per-call result. The submitted payload carries only what the SDK directly observed — transcript (with per-message speech timing so the platform can recompute WPM, talk-ratio and interruptions), recording, provider call data, terminal status. Start/end and duration are derived from the observed speech timestamps when the engine does not stamp case-level times. Recordings are streamed to the platform as a multipart upload. Submission is env-gated (FI_BASE_URL / FI_API_KEY / FI_SECRET_KEY / FI_RUN_TEST_ID); with any missing it records not_configured and no HTTP is attempted, so local runs are unaffected. Adds oss/simulation-acceptance/run_platform_voice_case.py to run one voice acceptance case end-to-end and submit its report through the sink.
Extends the FutureAGI sink to pull the agent-under-test's provider-reported usage from case evidence and fold it into provider_call_data under the normalized usage.llm shape the platform reads, plus costs.cost_cents. Provider-agnostic: dispatches to a per-provider extractor since each reports differently — Vapi costBreakdown (llmPrompt/CompletionTokens, dollar cost), Retell call_cost + llm_token_usage (combined_cost in cents, total-only or split tokens), LiveKit normalized usage. This is the target agent's real usage, not the FutureAGI simulator's. Timing and transcript already flow for every target because they come from the shared LiveKit session, not the provider evidence.
Author
|
Pushed two commits extending this branch for the platform-submission path (ALK → FutureAGI voice ingestion):
Backend counterpart: future-agi#1976. |
Requirement-gated metrics (browser/multi-agent/orchestration/voice/etc.
coverage + quality) early-return a vacuous 1.0 when a case configures no
requirement for them, inflating the flat-mean aggregate toward ~0.94
regardless of real agent quality.
Add an 'applicable' flag to AgentReportMetricResult, classify these
unconfigured metrics as not-applicable (reason ends 'provided.' /
'configured.' / 'not required.'), and score only applicable metrics in
_weighted_average. Genuine safety passes ('No secret-like output
detected.', 'No unsafe memory writes.') stay applicable and counted.
…ites Eval-suite providers previously supported only offline stubs (echo, scripted, artifact, python_callable). Add a litellm-backed provider so suites can call a real LLM directly: type 'vertex'/'gemini' (bare model, auto-prefixed vertex_ai/) for Vertex AI, or 'litellm' with a fully-qualified model string for any other provider. Wired at the single _provider_output choke point, so both 'agent-learn eval' and 'agent-learn optimize-eval' pick it up. Vertex auth via GOOGLE_APPLICATION_CREDENTIALS; routing via vertex_project/vertex_location provider fields or VERTEXAI_* env.
MetaPrompt, ProTeGi, and PromptWizard hardcoded the task generator (the model that runs candidate prompts while scoring) to gpt-4o-mini/gpt-5-mini, forcing an OpenAI key even when the teacher generator was another provider. Add a task_model constructor arg (default None) that falls back to the teacher generator's model, so passing e.g. a Vertex teacher makes the whole optimizer run on Vertex. Backward-compatible: pass task_model to override.
Evals: add `fi_eval` suite assertion type that scores case output with any hosted FutureAGI eval template via fi.evals.evaluate (platform/turing engine, FI_* creds), pass = score >= threshold. Thread case vars into assertions and expose public evaluate_assertions(). Optimizers: register curriculum/pareto/feedback tokens in _optimizer_cls, and add a generative eval-suite bridge (generative_suite.py) so gepa/protegi/ metaprompt/promptwizard/random_search/bayesian_search run real LLM prompt rewriting from an eval suite, scored against the suite's own assertions. optimize_eval_suite routes generative tokens before the deterministic target.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Nine commits that take fi.simulate from local-only, LiveKit-hard-coded runs to a full provider-agnostic acceptance surface (LiveKit + Vapi + Retell) with platform-driven scenario generation and prod-hardened SIP support. ~19k lines added / 2.3k removed across 111 files.
What lands
ee69932):fi.simulate.simulation.voice_promptwith call-direction-aware, role-locked persona policy;fi.alk.studio._generate(idempotent Agent Definition create/reuse, safe metadata upload, async Scenario poll + dataset-table pagination); scenario cache; runtime/artifacts/environments/evidence/results scaffolding.67033fa):agent-learn scenario generate, real dataset-table hydration in_download,TelephonyTransportwith E.164 validation andwait_until_answeredsemantics.c063404): endpoints/realtime/simulator/results namespaces, Vapi + Retell evidence adapters, LiveKit instrumentation skeleton,simulation/matrix.py+matrix_cli.py(surfaces per-leg status so evaluator score no longer masks failed runs), Vertex + Gemini backends.e16933a): skip prematureCreateRoomforsip_outbound; per-caseSIPDispatchRuleDirectbound toLIVEKIT_INBOUND_TRUNK_ID;AgentSessionlocks onto SIP caller viaparticipant_kinds=[SIP]+ identity; typed sanitized failure codes; grade natural target hang-ups as completed when both roles spoke.3cf8c8f): direct-target flow support.963a399): splitAgentDefinitioninto target-agent shape andLiveKitSimulatorRuntime; renamed primary Platform creds toFI_API_KEY/FI_SECRET_KEY/FI_BASE_URL(legacy aliases retained); scenario-generation-only platform registration excludes provider secrets and runtime creds.35d6525): studio fix.d2b1420): full sweep flow wiring.b53e4a7):LiveKitSimulatorRuntime.room_name_verbatimhonored by_resolve_room_nameso runs join a pre-existing dispatch rule bound to a fixed room (unblocks livekit-infra DID pool). Multi-persona runs guard against verbatim reuse.trigger_livekit_outboundshort-circuits room discovery onACCEPTANCE_ROOM_NAME_OVERRIDE.min_turn_messagesbecomes validation floor; empty/silent/short conversations now fail correctly; matrix runner non-zero on eval failure; split simulator utterances merged before turn-count.Test plan
pytest tests/runtime/test_livekit_engine.py tests/test_acceptance_regressions.py tests/test_voice_simulation.py tests/test_acceptance_run_voice_case.py tests/test_acceptance_trigger.py tests/test_acceptance_voice_cases.py— 73 passed.run_prod_livekit_matrix.py --case 1.1.1 --case 1.2.1 --use-pool) — both cases pass against production GKE LiveKit.agent-learn scenario generateagainst staging Platform (idempotent Agent Definition path).FI_API_KEY/FI_SECRET_KEYalias fallbacks (FUTURE_AGI_*,AGENT_LEARNING_*) still resolve for existing tooling.