diff --git a/skills/assemblyai/SKILL.md b/skills/assemblyai/SKILL.md index 81d7948..63306e2 100644 --- a/skills/assemblyai/SKILL.md +++ b/skills/assemblyai/SKILL.md @@ -43,36 +43,35 @@ Authorization: YOUR_API_KEY ## Speech-to-Text Models +**`universal-3-5-pro` is the model to use across the board.** It is GA for **both** realtime/streaming and pre-recorded/async transcription and is the default on each. Use it everywhere by default; drop to `universal-2` only for cost or for languages outside Universal-3.5 Pro's 18. `universal-3-pro` (async) and `u3-rt-pro` (streaming) have both been **superseded by `universal-3-5-pro`** and removed from their model lists/enums. + ### Pre-Recorded | Model | Languages | Best For | |-------|-----------|----------| -| **Universal-3.5 Pro** | 18 (auto-falls back to Universal-2 for the other 99) | Latest flagship: best accuracy, native code-switching across its 18 languages, contextual `prompt`, keyterms up to 1,000 words | -| **Universal-3 Pro** | 6 (en, es, de, fr, pt, it) | Highest accuracy, promptable transcription, keyterms up to 1,000 words | -| **Universal-2** | 99 | Broadest language coverage, keyterms up to 200 words | - -Use `speech_models` as a priority list with fallback: `["universal-3-pro", "universal-2"]` (the default when omitted). +| **Universal-3.5 Pro** (recommended default) | 18 (auto-falls back to Universal-2 for the other 99) | Flagship: best accuracy, fastest, native code-switching across its 18 languages, contextual `prompt`, keyterms up to 1,000 words | +| **Universal-2** | 99 | Cost-effective; broadest language coverage; keyterms up to 200 words; fallback for languages outside Universal-3.5 Pro's 18 | -**Universal-3.5 Pro is also available for pre-recorded/async transcription**, not just streaming. Opt in with `speech_models: ["universal-3-5-pro"]` on `POST /v2/transcript`. It supports contextual `prompt` (a plain-language *description* of the audio — domain/scenario/full detail) and `keyterms_prompt` (up to 1,000 terms), does native code-switching, and **auto-falls back to Universal-2** for languages outside its 18. Note: the formal OpenAPI `speech_models` enum still lists only `universal-3-pro`/`universal-2`, but the async API accepts `universal-3-5-pro`. +On `POST /v2/transcript`, `speech_models` is a priority list with fallback and **defaults to `["universal-3-5-pro", "universal-2"]`** when omitted — Universal-3.5 Pro handles its 18 languages and automatically falls back to Universal-2 for the rest. The async `speech_models` enum now accepts only `universal-3-5-pro` and `universal-2` (`universal-3-pro` has been removed, superseded by `universal-3-5-pro`). Universal-3.5 Pro supports contextual `prompt` (a plain-language *description* of the audio) and `keyterms_prompt` (up to 1,000 terms). Check the `speech_model_used` response field to see which model actually ran. ### Streaming | Model | Languages | Best For | |-------|-----------|----------| -| **universal-3-5-pro** | 18 | **Recommended default for new realtime/streaming code** — next-gen flagship: more languages, improved prompting + conversational context | -| **u3-rt-pro** | 6 | Universal-3 Pro Streaming — punctuation-based turn detection, promptable, `mode` accuracy/latency tradeoff | +| **universal-3-5-pro** | 18 | **Recommended default for new realtime/streaming code** — next-gen flagship: more languages, native code-switching, improved prompting + conversational context | | **universal-streaming-english** | 1 (English) | Voice agents, ~300ms latency | | **universal-streaming-multilingual** | 6 | Per-utterance language detection | +| **u3-rt-pro** | 6 | **Legacy** — Universal-3 Pro Streaming, **removed July 2026** from the model picker and streaming spec `speech_model` enum. Superseded by `universal-3-5-pro`. Still seen in older code | | **whisper-rt** | 99+ | **Legacy** — removed from the public model picker (June 2026) and the streaming spec enums, but still functional via `speech_model: whisper-rt` for broadest streaming language coverage, auto-detect only | -For realtime/streaming STT, use `speech_model: "universal-3-5-pro"` by default. The raw API parameter is optional and defaults to `universal-3-5-pro`; set it explicitly when pinning behavior or using an SDK that requires the field. The `mode` connection param (universal-3-5-pro / u3-rt-pro) trades off accuracy vs latency: `min_latency`, `balanced` (default), `max_accuracy`. +For realtime/streaming STT, use `speech_model: "universal-3-5-pro"` by default. The raw API parameter is optional and defaults to `universal-3-5-pro`; set it explicitly when pinning behavior or using an SDK that requires the field. The streaming spec `speech_model` enum now lists only `universal-3-5-pro`, `universal-streaming-english`, and `universal-streaming-multilingual`. The `mode` connection param (universal-3-5-pro only) trades off accuracy vs latency: `min_latency`, `balanced` (default), `max_accuracy`. ### Medical Mode (Add-On) `domain: "medical-v1"` enables Medical Mode — an add-on that improves accuracy for medical terminology (medications, procedures, conditions, dosages). Works with both pre-recorded and streaming models. -- **Pre-recorded:** Universal-3 Pro (`domain: "medical-v1"` in request body), Universal-2 -- **Streaming:** universal-3-5-pro, u3-rt-pro, universal-streaming-english, universal-streaming-multilingual +- **Pre-recorded:** Universal-3.5 Pro (`domain: "medical-v1"` in request body), Universal-2 +- **Streaming:** universal-3-5-pro, universal-streaming-english, universal-streaming-multilingual - **Supported languages:** English, Spanish, German, French (4 languages only) - Billed as a separate add-on. If used with an unsupported language, the API ignores `domain` and returns a warning — transcript still completes and you are NOT charged for Medical Mode. @@ -80,7 +79,7 @@ For realtime/streaming STT, use `speech_model: "universal-3-5-pro"` by default. `prompt` and `keyterms_prompt` are **complementary** — use either, or both together. Neither changes the output format, and both work the same way for **streaming and async** (`POST /v2/transcript`). Transcription behavior (verbatim, punctuation, formatting) is built in and managed by AssemblyAI. -- **`prompt`** (string, ~1500 chars max): a plain-language **description of the audio** — its domain, scenario, or full details. It carries *context*, **not instructions** — formatting/behavioral commands (punctuation rules, "transcribe verbatim", "don't…") are not supported and are ignored. The model stays grounded in the audio: irrelevant or only-partially-applicable context won't make it insert words that weren't spoken, so you can safely send the same description on every session/segment. +- **`prompt`** (string; length cap varies by API — streaming ≤1750 chars, async up to ~1,500 words, Sync STT ≤4096 chars): a plain-language **description of the audio** — its domain, scenario, or full details. It carries *context*, **not instructions** — formatting/behavioral commands (punctuation rules, "transcribe verbatim", "don't…") are not supported and are ignored. The model stays grounded in the audio: irrelevant or only-partially-applicable context won't make it insert words that weren't spoken, so you can safely send the same description on every session/segment. - **`keyterms_prompt`** (string[]): an explicit list of names/brands/domain terms to boost. Streaming: up to **100** terms, ≤50 chars each. Async: up to **1,000** terms. **Contextual prompt — three levels of specificity** (use the least specific that covers your case): @@ -129,11 +128,11 @@ See `references/llm-gateway.md` for models, tool calling, structured outputs, an | Gotcha | Details | |--------|---------| | `prompt` + `keyterms_prompt` | **Complementary** for Universal-3.5 Pro — use either or both together. `prompt` is a contextual *description* of the audio; `keyterms_prompt` is an explicit term list. Neither changes output formatting | -| `summarization` / `auto_chapters` | **Deprecated.** Use LLM Gateway instead (transcribe → send text to LLM) | +| `summarization` / `auto_chapters` (top-level params) | **Deprecated.** Use **Speech Understanding** `summarization` (chaptered summary w/ timestamps + headlines) or `action_items` — `speech_understanding.request.` on `POST /v2/transcript` — or the LLM Gateway for fully custom prompts | | PII redaction scope | Only redacts words in `text` — other feature outputs (entities, summaries) may still expose sensitive data | | Upload key scoping | Files uploaded with one API key project cannot be transcribed with a different project's key | | Structured outputs | Supported by OpenAI, Gemini, Claude 4.5+, Qwen, and Kimi — Claude 3.x does NOT support `json_schema` structured outputs | -| U3 Pro-family turn detection | `universal-3-5-pro` and `u3-rt-pro` use punctuation (`.` `?` `!`), NOT confidence thresholds — `end_of_turn_confidence_threshold` has no effect | +| universal-3-5-pro turn detection | `universal-3-5-pro` uses punctuation (`.` `?` `!`), NOT confidence thresholds — `end_of_turn_confidence_threshold` has no effect (it applies only to Universal Streaming English/Multilingual) | | `prompt` is context, not instructions | Universal-3.5 Pro's `prompt` *describes* the audio (domain/scenario/details). Formatting or behavioral commands (punctuation rules, "transcribe verbatim", negative directives like "don't…") are **not supported** and are ignored — transcription behavior is managed internally | | PII audio redaction method | `override_audio_redaction_method: "silence"` replaces PII with silence instead of default beep | | Language detection | Requires minimum 15 seconds of spoken audio for reliable results | @@ -154,9 +153,14 @@ See `references/llm-gateway.md` for models, tool calling, structured outputs, an | LLM Gateway `tool_calls` location | `tool_calls` lives at `choices[i].message.tool_calls` (under `message`), NOT at `choices[i].tool_calls` (under `choice`). `content` is `null` when only tool_calls are present | | LLM Gateway `finish_reason` is provider-native | Don't branch tool-calling loops on `finish_reason == "tool_calls"` — the Gateway passes the provider's value through, so **Claude returns `tool_use`/`end_turn`** (OpenAI returns `tool_calls`/`stop`). Detect a tool call by the **presence of `message.tool_calls`**, not by `finish_reason` | | Transcript `metadata.warnings` | The `Transcript` response now includes an optional `metadata` object. When present, `metadata.warnings` is an array of `{message}` objects describing issues processed during transcription (e.g. Medical Mode skipped due to unsupported language). `metadata` is omitted entirely when there is nothing to report | -| U3 Pro-family streaming context carryover | On by default — the model carries prior finalized turns forward as context (per-session, ~3 entries, ~1500 chars). Pass your agent's spoken reply via `agent_context` (connection-time query param to seed an opening greeting, or mid-stream via `UpdateConfiguration`) so the model knows the question the user is answering. Use with `universal-3-5-pro` or `u3-rt-pro` | +| universal-3-5-pro streaming context carryover | On by default — the model carries prior finalized turns forward as context (per-session, up to `previous_context_n_turns` entries, server default `5`, range 0–100). Pass your agent's spoken reply via `agent_context` (connection-time query param to seed an opening greeting, or mid-stream via `UpdateConfiguration`; ≤1750 chars per value) so the model knows the question the user is answering. `universal-3-5-pro` only | | Streaming diarization revised labels | With `speaker_labels` enabled, a single `SpeakerRevision` message is emitted right before `Termination` (after you send `Terminate`), containing a `revisions` array of only the turns whose speaker labels changed (matched by `turn_order`). Text and word timestamps never change — only speaker assignments. Adds ~400ms latency at session close. Use it for the final, highest-quality attribution | | LLM Gateway `model_region: "global"` | Optional request field (only accepted value `"global"`) routes to the provider's global, non-region endpoints for lower cost. Live for Anthropic Claude now; Google Gemini 3 series coming soon. Omit for default in-region processing. **Effective July 1, 2026, in-region LLM Gateway requests cost 10% more** (provider pass-through, no AssemblyAI upcharge) — global routing keeps current pricing | +| Streaming language selection | The connection param is **`language_codes`** (plural, a **list** e.g. `["en","es"]`) — there is no singular `language_code` *input* param. It steers output toward those languages while still allowing native code-switching. `universal-3-5-pro` only. (`language_code` appears only as an *output* field when `language_detection: true`.) | +| Streaming Opus input | `encoding` accepts `opus` (raw packets — one per binary WS message) and `ogg_opus` (Ogg stream — arbitrary chunks; ffmpeg/gstreamer/MediaRecorder output), in addition to `pcm_s16le`/`pcm_mulaw`. For Opus, `sample_rate` is ignored (stream is self-describing) | +| Voice Agent HTTP tool `headers` | Server-side HTTP tools take `headers` as a **list of `{name, value}` entries** (since June 2026), NOT a `{name: value}` dict. Reads return `{name, last_set_at}` only — values are write-only and never returned (not masked as `"***"`) | +| Voice Agent BYO LLM (`llm`) | Set `llm` on a **stored agent** (REST create/update, not `session.update`) to run on your own OpenAI-compatible endpoint: `[{base_url, model, api_key}]` — one entry only, must stream, `api_key` write-only. `"llm": []` reverts to the managed model. Point `base_url` at the LLM Gateway to use a frontier model on your AssemblyAI account | +| Voice Agent webhooks | Subscribe to Voice Agent events via `POST https://agents.assemblyai.com/v1/webhook-subscriptions` (events: `session.started`/`session.completed`/`call.connected`/`call.ended`/`call.failed`; scope with `agent_id` or omit for account-wide). See `references/api-reference.md` | ## Common Mistakes @@ -164,7 +168,7 @@ See `references/llm-gateway.md` for models, tool calling, structured outputs, an |---------|------------| | `Authorization: Bearer KEY` | `Authorization: KEY` (no Bearer prefix) — BUT the Voice Agent API (`agents.assemblyai.com`) uses `Authorization: Bearer KEY` | | Using LeMUR API | **Deprecated.** Use LLM Gateway instead | -| Using `summarization` or `auto_chapters` | **Deprecated.** Use LLM Gateway instead (transcribe then summarize via LLM) | +| Using top-level `summarization` or `auto_chapters` | **Deprecated.** Use Speech Understanding `summarization`/`action_items` (`speech_understanding.request.`) or the LLM Gateway | | LeMUR `transcript_ids` with LLM Gateway | Pass transcript text in messages, not IDs | | `anthropic/claude-...` model IDs | No provider prefix: `claude-sonnet-4-5-20250929` not `anthropic/claude-sonnet-4-5-20250929` | | `claude-opus-4-20250514` / `claude-sonnet-4-20250514` on LLM Gateway | **Removed June 2026.** Use Claude Opus 4.5/4.6/4.7 or Claude Sonnet 4.5/4.6 | @@ -172,9 +176,9 @@ See `references/llm-gateway.md` for models, tool calling, structured outputs, an | Using Java/Go/C# SDKs | **Discontinued.** Use Python, JS/TS, Ruby, or raw API | | `word_boost` on the async REST API | Use `keyterms_prompt` instead. **Exception:** the Sync STT API *does* use `word_boost` (in its `config` part) — that's its documented keyterms param | | Hardcoding v2 streaming URL | v3 (`/v3/ws`) is current; v2 still works but is legacy | -| Assuming streaming defaults to `u3-rt-pro` | Streaming `speech_model` defaults to `universal-3-5-pro` at the raw API layer. Set a different model only when you intentionally need legacy behavior, cost tradeoffs, or broader language coverage | +| Using `speech_model=u3-rt-pro` for streaming | **Removed July 2026** from the model picker and streaming spec enum — superseded by `universal-3-5-pro` (the streaming default). Set a different model only for cost tradeoffs (`universal-streaming-english`/`-multilingual`) | | Python SDK rejects `universal-3-5-pro` | Upgrade to `assemblyai>=0.64.21` for Streaming v3 SDK support. Older SDKs such as `0.64.4` validate `speech_model` against an enum that omits `universal-3-5-pro` | -| `aai.SpeechModel.universal_3_pro` in Python SDK | Use raw strings: `"universal-3-pro"`, `"universal-2"` — these enum aliases don't exist in the SDK | +| `aai.SpeechModel.universal_3_5_pro` in Python SDK | Use raw strings: `"universal-3-5-pro"`, `"universal-2"` — these enum aliases don't exist in the SDK | | S2S `session.update` without `"session"` key | Must wrap config: `{"type":"session.update","session":{...}}` | | S2S tool schema using `{"function":{...}}` nesting | S2S tools are flat: `{"type":"function","name":"...","description":"...","parameters":{...}}` | | Voice Agent S2S URL | Correct URL: `wss://agents.assemblyai.com/v1/ws` — not `/v1/voice` (renamed April 2026), `/v1/realtime` (older), or `speech-to-speech.us.assemblyai.com` (very old) | @@ -197,9 +201,9 @@ Read the relevant reference file based on what the user needs: | `references/streaming.md` | Real-time/streaming STT, v3 protocol, temp tokens, error codes | | `references/voice-agents.md` | Voice agent integrations: LiveKit, Pipecat, turn detection, latency optimization | | `references/llm-gateway.md` | Applying LLMs to transcripts, tool calling, available models | -| `references/speech-understanding.md` | Translation, speaker identification, custom formatting | +| `references/speech-understanding.md` | Translation, speaker identification, custom formatting, summarization, action items | | `references/audio-intelligence.md` | PII redaction, diarization, summarization, sentiment, chapters | -| `references/api-reference.md` | Full parameter list, export endpoints, webhooks, upload, PII policies, Sync STT API, Voice Agents REST API (stored agents) | +| `references/api-reference.md` | Full parameter list, export endpoints, webhooks, upload, PII policies, Sync STT API, Voice Agents REST API (stored agents, HTTP-tool headers, BYO LLM, webhook subscriptions) | ## API Spec Source of Truth diff --git a/skills/assemblyai/references/api-reference.md b/skills/assemblyai/references/api-reference.md index ede86f4..b928ae3 100644 --- a/skills/assemblyai/references/api-reference.md +++ b/skills/assemblyai/references/api-reference.md @@ -32,7 +32,7 @@ Submit an audio file for transcription. Send a JSON body with the parameters bel | Parameter | Type | Description | |---|---|---| | `audio_url` | string | **Required.** URL of the audio file to transcribe. Can be a public URL or an `upload_url` from the upload endpoint. | -| `speech_models` | array | **Optional** (as of June 2026). Priority-ordered list of speech models to use (e.g., `["universal-3-5-pro", "universal-2"]`). First model is used if supported; falls back to next. If omitted, defaults to `["universal-3-pro", "universal-2"]`. Universal-3.5 Pro is accepted here (`["universal-3-5-pro"]`). | +| `speech_models` | array | **Optional.** Priority-ordered list of speech models. First supported model is used; falls back to the next. **If omitted, defaults to `["universal-3-5-pro", "universal-2"]`.** The enum now accepts only `universal-3-5-pro` and `universal-2` — `universal-3-pro` was removed (superseded by `universal-3-5-pro`). The `speech_model_used` response field reports which model actually ran. | | `prompt` | string | For Universal-3.5 Pro, a contextual *description* of the audio (domain → scenario → full detail), **not** formatting/behavioral instructions (those are ignored). **Complementary with `keyterms_prompt`** — both can be set together. | | `keyterms_prompt` | array | List of key terms/phrases (strings) to boost recognition accuracy — up to **1000** terms for Universal-3.5 Pro, **200** for Universal-2, max **6 words per phrase**. **Complementary with `prompt`** — both can be set together. | | `language_code` | string | Language code (e.g., `"en_us"`, `"es"`, `"fr"`). Defaults to `"en_us"`. | @@ -42,11 +42,11 @@ Submit an audio file for transcription. Send a JSON body with the parameters bel | `speaker_labels` | boolean | Enable speaker diarization. Default `false`. | | `sentiment_analysis` | boolean | Enable sentiment analysis on each sentence. Default `false`. | | `entity_detection` | boolean | Enable entity detection. Default `false`. | -| `auto_chapters` | boolean | **Deprecated.** Enable auto chapters. Use LLM Gateway instead. | +| `auto_chapters` | boolean | **Deprecated.** Use Speech Understanding `summarization` (chaptered summary; see `speech-understanding.md`) or the LLM Gateway instead. | | `iab_categories` | boolean | Enable IAB content category detection. Default `false`. | | `content_safety` | boolean | Enable content safety detection. Default `false`. | | `content_safety_confidence` | integer | Minimum confidence threshold (25-100) for content safety labels. | -| `summarization` | boolean | **Deprecated.** Enable summarization. Use LLM Gateway instead. | +| `summarization` | boolean | **Deprecated.** Use Speech Understanding `summarization`/`action_items` (see `speech-understanding.md`) or the LLM Gateway instead. | | `summary_model` | string | **Deprecated.** Model for summarization. | | `summary_type` | string | **Deprecated.** Type of summary. | | `redact_pii` | boolean | Enable PII redaction. Default `false`. | @@ -68,10 +68,10 @@ Submit an audio file for transcription. Send a JSON body with the parameters bel | `speech_understanding` | object | Enable Speech Understanding inline. Features nest under `speech_understanding.request` (the `request` wrapper is required): `translation`, `speaker_identification`, and/or `custom_formatting`. See `speech-understanding.md`. | | `speakers_expected` | integer | Hint for number of speakers (diarization). Deprecated in favor of `speaker_options`. | | `speaker_options` | object | Diarization options: `min_speakers_expected` (int, default 1), `max_speakers_expected` (int). | -| `temperature` | float | 0–1. Controls randomness. Universal-3 Pro only. | -| `domain` | string | Domain-specific model variant. `"medical-v1"` enables Medical Mode (EN, ES, DE, FR). Supported on Universal-3 Pro and Universal-2. | -| `remove_audio_tags` | string | Remove inline annotations from the transcript. `"all"` removes all (audio event markers and speaker cues); `"speaker"` removes only speaker cues while keeping other annotations. Universal-3 Pro only. | -| `language_codes` | array | List of language codes for code-switching (must include `"en"`). Universal-3 Pro only. | +| `temperature` | float | 0–1. Controls randomness. Universal-3.5 Pro only. | +| `domain` | string | Domain-specific model variant. `"medical-v1"` enables Medical Mode (EN, ES, DE, FR). Supported on Universal-3.5 Pro and Universal-2. | +| `remove_audio_tags` | string | Remove inline annotations from the transcript. `"all"` removes all (audio event markers and speaker cues); `"speaker"` removes only speaker cues while keeping other annotations. Universal-3.5 Pro only. | +| `language_codes` | array | List of language codes for code-switching (must include `"en"`). Universal-3.5 Pro only. | | `audio_start_from` | integer | Start transcription from this time offset, in **milliseconds**. | | `audio_end_at` | integer | End transcription at this time offset, in **milliseconds**. | | `speech_threshold` | float | Confidence threshold (0-1) for filtering low-confidence speech. Requires at least **30 seconds** of audio. | @@ -105,7 +105,7 @@ The transcript response may include an optional `metadata` object with additiona "metadata": { "domain_used": null, "warnings": [ - { "message": "'ja' is not supported in universal-3-pro — transcription is handled by universal-2. To silence this warning, set speech_models: [\"universal-3-pro\", \"universal-2\"]." } + { "message": "'ur' is not supported in universal-3-5-pro — transcription is handled by universal-2. To silence this warning, set speech_models: [\"universal-3-5-pro\", \"universal-2\"]." } ] } } @@ -479,11 +479,11 @@ A REST API for creating **reusable** voice agents. An agent stores its `system_p |---------------|-------------| | `POST /v1/agents` | Create an agent. Returns `201` with the full record including a generated `id`. | | `GET /v1/agents` | List agents (lightweight records, no tools/prompt), newest first. | -| `GET /v1/agents/{agent_id}` | Retrieve one agent. Tool header **values** are masked as `"***"`. | +| `GET /v1/agents/{agent_id}` | Retrieve one agent. Read responses return tool-header **names** and `last_set_at` only — header **values are never returned** (write-only, encrypted at rest; they are *not* masked as `"***"`). Likewise a connected LLM (`llm`) comes back as `base_url` + `model` only, never `api_key`. | | `PUT /v1/agents/{agent_id}` | Update an agent. Every field optional — send only what you want to change. | | `DELETE /v1/agents/{agent_id}` | Delete an agent. Returns `204`, no body. | -**Create body** (`application/json`): required `name`, `system_prompt`, `voice`; optional `greeting`, `input`, `output`, `tools`. Note `voice` is a **top-level** field here (in the WebSocket `session.update` it lives under `output.voice`). `greeting` is spoken straight to TTS on connect — omit it to have the agent listen first. +**Create body** (`application/json`): required `name`, `system_prompt`, `voice`; optional `greeting`, `input`, `output`, `tools`, `llm`. Note `voice` is a **top-level** field here (in the WebSocket `session.update` it lives under `output.voice`). `greeting` is spoken straight to TTS on connect — omit it to have the agent listen first. ```bash curl -X POST https://agents.assemblyai.com/v1/agents \ @@ -491,3 +491,43 @@ curl -X POST https://agents.assemblyai.com/v1/agents \ -H 'Content-Type: application/json' \ -d '{"name":"Support Bot","system_prompt":"You are a concise support agent.","voice":"ivy","greeting":"Hi, how can I help?"}' ``` + +### Server-side HTTP tools (`headers` shape) + +A tool with an `http` block (`url`, `http_method`, `headers`) is executed **server-side** — AssemblyAI makes the request on your behalf. As of June 2026 `headers` is a **list of `{name, value}` entries**, NOT a `{name: value}` dict: + +```json +{ "http": { "url": "https://api.example.com/orders", "http_method": "POST", + "headers": [ { "name": "Authorization", "value": "Bearer xyz" } ] } } +``` + +- **Write (create/update):** each entry is `{ name, value?, remove? }`. Provide `value` to set/rotate; provide name only to keep the stored value unchanged (round-tripping on update); `remove: true` deletes it. Sending both `value` and `remove: true` is rejected. +- **Read (get/list):** each entry is `{ name, last_set_at }` — values are never returned. +- Server-side constraints: `https` + public hosts only (private/loopback/CGNAT blocked); redirects not followed; response body capped at 8 KiB; per-call timeout from `timeout_seconds`. + +### Bring your own LLM (`llm`) + +Point an agent at your own **OpenAI-compatible** chat-completions endpoint instead of AssemblyAI's managed model. Set the `llm` field (a list; **only one entry accepted today**) on create/update: + +```json +{ "llm": [ { "base_url": "https://api.openai.com/v1", "model": "gpt-4o-mini", "api_key": "sk-..." } ] } +``` + +- `base_url` (required, `https` + public host — the agent calls `POST {base_url}/chat/completions`), `model` (required), `api_key` (required, **write-only**, never returned). +- Must support **streamed** chat completions. Send `"llm": []` to switch back to the managed model. +- To use a frontier model without your own provider account, point `base_url` at the **LLM Gateway** (`https://llm-gateway.assemblyai.com/v1`, EU: `https://llm-gateway.eu.assemblyai.com/v1`) and pass your AssemblyAI key as `api_key`. + +## 18. Voice Agent Webhooks API + +A REST API (base URL `https://agents.assemblyai.com`, same auth) to subscribe URLs to Voice Agent lifecycle events. + +| Method & Path | Description | +|---------------|-------------| +| `POST /v1/webhook-subscriptions` | Subscribe a URL to one or more events. Scope to one agent with `agent_id`, or omit for account-wide events. Returns the created subscription. | +| `GET /v1/webhook-subscriptions` | List subscriptions (paginated). | +| `GET /v1/webhook-subscriptions/{subscription_id}` | Retrieve one subscription. | +| `PUT /v1/webhook-subscriptions/{subscription_id}` | Update `url`, `events`, `secret`, or `enabled`. Sending a new `secret` rotates it and increments `secret_version`. | +| `DELETE /v1/webhook-subscriptions/{subscription_id}` | Delete a subscription. Returns `204`. | + +- **Events:** `session.started`, `session.completed`, `call.connected`, `call.ended`, `call.failed`. +- **Body fields:** `url` (`https` + public host, required), `events` (array, required), `secret` (32–256 printable ASCII chars, no whitespace — signing secret, **write-only**, only its `secret_version` is returned), `agent_id` (optional; omit for account-wide). diff --git a/skills/assemblyai/references/audio-intelligence.md b/skills/assemblyai/references/audio-intelligence.md index dfc342d..1254753 100644 --- a/skills/assemblyai/references/audio-intelligence.md +++ b/skills/assemblyai/references/audio-intelligence.md @@ -77,7 +77,7 @@ Legacy parameters (still functional but not recommended for new code): - Enable with `domain: "medical-v1"` in the request body - Improves accuracy for medical terminology: medications, procedures, conditions, dosages -- Supported models: Universal-3 Pro, Universal-2 (pre-recorded); all streaming models +- Supported models: Universal-3.5 Pro, Universal-2 (pre-recorded); all streaming models - Supported languages: English, Spanish, German, French only - Billed as a separate add-on - If used with an unsupported language, the API ignores `domain` and returns a warning — no charge is applied diff --git a/skills/assemblyai/references/python-sdk.md b/skills/assemblyai/references/python-sdk.md index 31d95a1..03655f8 100644 --- a/skills/assemblyai/references/python-sdk.md +++ b/skills/assemblyai/references/python-sdk.md @@ -47,7 +47,7 @@ Use `speech_models` to specify a preferred model with automatic fallback: ```python config = aai.TranscriptionConfig( - speech_models=["universal-3-pro", "universal-2"] + speech_models=["universal-3-5-pro", "universal-2"] ) transcriber = aai.Transcriber(config=config) diff --git a/skills/assemblyai/references/speech-understanding.md b/skills/assemblyai/references/speech-understanding.md index 8f9ad5b..33aa4cf 100644 --- a/skills/assemblyai/references/speech-understanding.md +++ b/skills/assemblyai/references/speech-understanding.md @@ -2,7 +2,9 @@ ## Overview -Speech Understanding provides post-transcription intelligence: **Translation, Speaker Identification, and Custom Formatting**. (Entity Detection, Sentiment Analysis, Key Phrases, and Topic Detection are *not* part of this object — they remain classic boolean transcript params; see `audio-intelligence.md`.) +Speech Understanding provides post-transcription intelligence: **Translation, Speaker Identification, Custom Formatting, Summarization, and Action Items**. (Entity Detection, Sentiment Analysis, Key Phrases, and Topic Detection are *not* part of this object — they remain classic boolean transcript params; see `audio-intelligence.md`.) + +> **Summarization vs the deprecated `summarization`/`auto_chapters` params:** the top-level `summarization` and `auto_chapters` boolean params on `POST /v2/transcript` are **deprecated**. The modern replacements are the Speech Understanding `summarization` (chaptered summary with timestamps + headlines) and `action_items` features below — or the LLM Gateway for fully custom prompts. Two ways to run it: @@ -29,7 +31,7 @@ Auth header: `Authorization: API_KEY` (no `Bearer` prefix). Translates the transcript into one or more target languages. -- **Models:** Universal-3 Pro and Universal-2 only. **Regions:** US & EU. Supports 100+ language codes. +- **Models:** Universal-3.5 Pro and Universal-2 only. **Regions:** US & EU. Supports 100+ language codes. ### Parameters (`speech_understanding.request.translation`) @@ -181,6 +183,93 @@ The format params are **strings (format patterns), not booleans**: `speech_understanding.response.custom_formatting` contains `formatted_text`, `formatted_utterances` (only when `format_utterances: true`), and a `mapping` object (original → formatted). +## Summarization + +Generates a chaptered summary of the transcript — each chapter has timestamps, a `text` summary, and a `headline`. Replaces the deprecated top-level `summarization`/`auto_chapters` params. **Open beta.** Models: `universal-3-5-pro`, `universal-2`. Regions: US & EU. + +### Parameters (`speech_understanding.request.summarization`) + +- `summary_type` (string, **required**): `"paragraph"` (longer, more detailed) or `"bullets"` (short, concise). +- `effort` (string, default `"low"`): `"low"` or `"medium"`. `medium` spends more processing power for higher-quality summaries — use it for high-stakes meetings, multilingual audio, or very long audio (≈1.5h+). Default `low` is sufficient for most cases. + +### Example + +```json +{ + "audio_url": "https://example.com/audio.mp3", + "speech_understanding": { + "request": { + "summarization": { "summary_type": "paragraph", "effort": "low" } + } + } +} +``` + +### Response + +`speech_understanding.response.summarization` → `{ status, summary_type, effort, summary: [{ start, end, text, headline }] }`: + +```json +{ + "speech_understanding": { + "response": { + "summarization": { + "status": "success", + "summary_type": "paragraph", + "effort": "low", + "summary": [ + { "start": 240, "end": 37100, "text": "Smoke from hundreds of Canadian wildfires...", "headline": "Smoke from Canadian Wildfires Affects US Air Quality" } + ] + } + } + } +} +``` + +## Action Items + +Extracts action items from the transcript, each with the source `quote` and a `timestamp`. **Open beta.** Models: `universal-3-5-pro`, `universal-2`. Regions: US & EU. + +### Parameters (`speech_understanding.request.action_items`) + +- `include_decisions` (boolean, default `false`): when `true`, also captures decisions made in the conversation as action items. +- `effort` (string, default `"low"`): `"low"` or `"medium"` (see Summarization for guidance on `medium`). + +Pass an empty object (`"action_items": {}`) to use defaults. + +### Example + +```json +{ + "audio_url": "https://example.com/audio.mp3", + "speech_understanding": { + "request": { + "action_items": { "include_decisions": true } + } + } +} +``` + +### Response + +`speech_understanding.response.action_items` → `{ status, effort, items: [{ action_item, quote, timestamp }] }`: + +```json +{ + "speech_understanding": { + "response": { + "action_items": { + "status": "success", + "effort": "low", + "items": [ + { "action_item": "Observe the expansion and contraction of the Arctic ice cap throughout the seasons.", "quote": "The Arctic ice cap ... expands in winter and contracts in summer.", "timestamp": 9520 } + ] + } + } + } +} +``` + ## Post-hoc via the Understanding Endpoint To run Speech Understanding on an **existing** transcript, POST to `/v1/understanding` with a `transcript_id` (same `speech_understanding.request.` structure): diff --git a/skills/assemblyai/references/streaming.md b/skills/assemblyai/references/streaming.md index 5358d52..e42e3c7 100644 --- a/skills/assemblyai/references/streaming.md +++ b/skills/assemblyai/references/streaming.md @@ -18,32 +18,32 @@ For new realtime/streaming code, use **`speech_model=universal-3-5-pro`** by def | Parameter | Description | |-----------|-------------| -| `speech_model` | **Optional at the raw API layer; default `universal-3-5-pro`.** Other models: `u3-rt-pro`, `universal-streaming-english`, `universal-streaming-multilingual`. `whisper-rt` (99+ languages) is **legacy** — removed from the public model picker and the streaming spec enums (June 2026) but still functional via `speech_model=whisper-rt` | -| `mode` | **universal-3-5-pro / u3-rt-pro only.** Accuracy/latency tradeoff: `min_latency` (fastest time-to-text), `balanced` (**default** — best for voice agents), or `max_accuracy` (highest accuracy, for scribes/post-call). Sets the per-mode defaults for `min_turn_silence`, `max_turn_silence`, `interruption_delay`, `continuous_partials`, and `vad_threshold`. Set at connection time and updatable mid-stream via `UpdateConfiguration`. | +| `speech_model` | **Optional at the raw API layer; default `universal-3-5-pro`.** Other current models: `universal-streaming-english`, `universal-streaming-multilingual`. The streaming spec `speech_model` enum now lists only these three. Two models are **legacy** — removed from the model picker and the spec enum but still seen in older code: `u3-rt-pro` (Universal-3 Pro Streaming, removed July 2026, superseded by `universal-3-5-pro`) and `whisper-rt` (99+ languages, removed June 2026, still functional via `speech_model=whisper-rt` for broadest language coverage). New integrations should use `universal-3-5-pro`. | +| `mode` | **universal-3-5-pro only.** Accuracy/latency tradeoff: `min_latency` (fastest time-to-text), `balanced` (**default** — best for voice agents), or `max_accuracy` (highest accuracy, for scribes/post-call). Sets the per-mode defaults for `min_turn_silence`, `max_turn_silence`, `interruption_delay`, `continuous_partials`, and `vad_threshold`. Set at connection time and updatable mid-stream via `UpdateConfiguration`. | | `sample_rate` | Audio sample rate in Hz (e.g., 16000) | -| `encoding` | Audio encoding: `pcm_s16le` or `pcm_mulaw` | -| `end_of_turn_confidence_threshold` | Confidence threshold for turn detection. Only affects Universal Streaming, not U3 Pro. **Officially deprecated** — tune `min_turn_silence`/`max_turn_silence` instead. | +| `encoding` | Audio encoding, default `pcm_s16le`. Raw PCM: `pcm_s16le` or `pcm_mulaw`. **Opus (added June 2026):** `opus` (raw Opus packets — each binary WebSocket message must contain exactly one packet) or `ogg_opus` (Ogg-encapsulated Opus stream, as produced by ffmpeg/gstreamer/opusenc/browser MediaRecorder — binary messages can be arbitrary chunks). For both Opus encodings `sample_rate` is ignored (the stream is self-describing). | +| `end_of_turn_confidence_threshold` | Confidence threshold for turn detection. Only affects Universal Streaming (English/Multilingual), not universal-3-5-pro. **Officially deprecated** — tune `min_turn_silence`/`max_turn_silence` instead. | | `format_turns` | Set to `true` to enable formatted final transcripts with punctuation, casing, and inverse text normalization (dates, times, phone numbers). Also activates turn-level keyterm boosting for Universal Streaming models. **Does NOT control digit rendering** — numerals (e.g. "22") are a model behavior, and lexical number output (e.g. "twenty-two") is not supported in streaming. | -| `prompt` | **universal-3-5-pro / u3-rt-pro only.** Natural-language *context about the audio* (domain, topic, scenario, conversation details) — **NOT** behavioral/formatting instructions. The transcription instruction (verbatim behavior, punctuation, formatting) is built in and managed by AssemblyAI; formatting or behavioral commands placed in `prompt` are not supported. **Complementary with `keyterms_prompt`** — use either or both together. If omitted, a built-in default prompt optimized for turn detection is used automatically. Recommended: test with no prompt first, then add context only for domain vocabulary the model gets wrong. | -| `keyterms_prompt` | JSON-encoded array of strings (up to 100 terms, max 50 chars each) to bias transcription (universal-3-5-pro, u3-rt-pro, and Universal Streaming). **Complementary with `prompt`** — both can be set together. When passing via URL query param, must be JSON.stringify'd: `keyterms_prompt=["term1","term2"]`. Costs additional $0.04/hr. | +| `prompt` | **universal-3-5-pro only.** Max **1750 characters.** Natural-language *context about the audio* (domain, topic, scenario, conversation details) — **NOT** behavioral/formatting instructions. The transcription instruction (verbatim behavior, punctuation, formatting) is built in and managed by AssemblyAI; formatting or behavioral commands placed in `prompt` are not supported. **Complementary with `keyterms_prompt`** — use either or both together. If omitted, a built-in default prompt optimized for turn detection is used automatically. Recommended: test with no prompt first, then add context only for domain vocabulary the model gets wrong. | +| `keyterms_prompt` | JSON-encoded array of strings (up to 100 terms, max 50 chars each) to bias transcription (universal-3-5-pro and Universal Streaming). **Complementary with `prompt`** — both can be set together. When passing via URL query param, must be JSON.stringify'd: `keyterms_prompt=["term1","term2"]`. Costs additional $0.04/hr. | | `inactivity_timeout` | Seconds of silence before session auto-closes | | `speaker_labels` | Enable diarization (`true`/`false`) | -| `max_speakers` | Maximum number of speakers for diarization | +| `max_speakers` | Integer 1–10. A **hard cap** (strict limit, not a hint) on speaker labels — once reached, additional speakers are merged into the closest existing label rather than given a new one. Give a little headroom above the expected count; setting it too high causes over-splitting. Only used when `speaker_labels` is enabled. | | `domain` | Set to `"medical-v1"` to enable Medical Mode (improves accuracy for medical terminology). Supported models: all streaming models. Supported languages: en, es, de, fr. | | `redact_pii` | Enable real-time PII redaction. Default `false`. Only applies to **final turns**. See Streaming PII Redaction below. | | `redact_pii_policies` | PII entity types to redact. Pass a comma-separated string (e.g. `person_name,phone_number`) over the raw WebSocket or an array via the SDK. Default: all. | | `redact_pii_sub` | Replacement scheme: `hash` (default — replaces with `#` chars) or `entity_name` (replaces with `[ENTITY_TYPE]`). | | `include_partial_turns` | Whether to include partial (non-final) turns. Defaults to `true` normally, but **`false` automatically** when `redact_pii` is `true` so unredacted text never reaches the client. | | `filter_profanity` | Filter profanity from transcripts (replaces with `***`). Default `false`. | -| `interruption_delay` | **universal-3-5-pro / u3-rt-pro only.** Integer milliseconds (0–1000, default `500`). How soon the first partial is emitted — lower = faster TTFT and earlier barge-in but more false interruptions; higher = more confident interruptions but slower partials. The server adds a minimum of ~256–300ms on top (`interruption_delay: 0` → ~256–300ms effective, `500` → ~756–800ms effective). The LiveKit plugin keeps the API default of `500`. | -| `continuous_partials` | **universal-3-5-pro / u3-rt-pro only.** Boolean — **default `true`** (changed June 2026; previously `false`). Now defaults to `true` on both the API directly and the LiveKit plugin. When `true`, emits additional partial transcripts approximately every ~3 seconds during long turns, each covering the full turn transcript so far. The first early partial (at 750ms / your `interruption_delay`) is unaffected. Set `false` if you only want silence-based partials. | -| `agent_context` | **universal-3-5-pro / u3-rt-pro only.** String (≤~1500 chars). Your voice agent's most recent spoken reply (TTS text), used as context for the next user turn — see Context Carryover below. Set at connection time to seed an opening greeting, and/or update mid-stream via `UpdateConfiguration`. | -| `previous_context_n_turns` | **universal-3-5-pro / u3-rt-pro only.** Integer (default `3`). Max number of prior conversation entries (finalized user transcripts plus any `agent_context` values) carried forward as context for each transcription. Set to `0` to disable automatic context carryover entirely. Most integrations leave this at the default — see Context Carryover below. | -| `vad_threshold` | **universal-3-5-pro / u3-rt-pro only.** Float 0.0–1.0 (default `0.3`). Confidence threshold for classifying audio frames as silence. Increase in noisy environments to reduce false speech detection. | -| `voice_focus` | **universal-3-5-pro / u3-rt-pro only.** Noise suppression that isolates the primary voice and suppresses background chatter, keyboard clicks, fan hum, and room echo before audio reaches the model. Set to `near-field` (headsets, handsets, close-talking mics) or `far-field` (conference rooms, laptop/drive-thru mics, distant capture). Omit to disable. Set as a connection parameter. | -| `voice_focus_threshold` | **universal-3-5-pro / u3-rt-pro only.** Optional float `0.0`–`1.0` controlling how aggressively background audio is suppressed when `voice_focus` is set — higher = more aggressive. | -| `language_code` | **universal-3-5-pro / u3-rt-pro only.** Optional ISO 639-1 code that biases the model toward a single language when you know the session is monolingual (improves language accuracy). Omit to keep default multilingual code-switching. `universal-3-5-pro` supports en, es, de, fr, pt, it, tr, nl, sv, no, da, fi, hi, vi, ar, he, ja, zh; `u3-rt-pro` supports en, es, fr, de, it, pt. | -| `language_detection` | **universal-3-5-pro / u3-rt-pro only.** Boolean (default `false`). When `true`, each `Turn` message includes the detected `language_code` and `language_confidence`. U3 Pro-family models natively code-switch without this — use it only when you need the per-turn language reported. | +| `interruption_delay` | **universal-3-5-pro only.** Integer milliseconds (0–1000). Default is **mode-dependent** (set by the `mode` preset), not a fixed value. How soon the first partial is emitted — lower = faster TTFT and earlier barge-in but more false interruptions; higher = more confident interruptions but slower partials. The server adds a **fixed 256ms** on top (`interruption_delay: 0` → 256ms effective, `500` → 756ms effective). | +| `continuous_partials` | **universal-3-5-pro only.** Boolean — **default `true`** (changed June 2026; previously `false`). Now defaults to `true` on both the API directly and the LiveKit plugin. When `true`, emits additional partial transcripts approximately every ~3 seconds during long turns, each covering the full turn transcript so far. The first early partial (timed by `interruption_delay`) is unaffected. When `speaker_labels` is enabled, continuous partials are disabled by default. Set `false` if you only want silence-based partials. | +| `agent_context` | **universal-3-5-pro only.** String (≤1750 chars per value). Your voice agent's most recent spoken reply (TTS text), used as context for the next user turn — see Context Carryover below. Set at connection time to seed an opening greeting, and/or update mid-stream via `UpdateConfiguration`. | +| `previous_context_n_turns` | **universal-3-5-pro only.** Integer, range 0–100 (server default currently `5`). Max number of prior conversation entries (finalized user transcripts plus any `agent_context` values) carried forward as context for each transcription. Set to `0` to disable automatic context carryover entirely. Most integrations leave this unset — see Context Carryover below. | +| `vad_threshold` | Float 0.0–1.0. Confidence threshold for classifying audio frames as silence — frames below this are considered silent. Increase in noisy environments to reduce false speech detection. Defaults: `0.2` on universal-3-5-pro (`0.5` when `speaker_labels` is enabled), `0.4` on Universal Streaming. | +| `voice_focus` | **universal-3-5-pro only.** Noise suppression that isolates the primary voice and suppresses background chatter, keyboard clicks, fan hum, and room echo before audio reaches the model. Set to `near-field` (headsets, handsets, close-talking mics) or `far-field` (conference rooms, laptop/drive-thru mics, distant capture). Omit to disable. Set as a connection parameter. | +| `voice_focus_threshold` | **universal-3-5-pro only.** Optional float `0.0`–`1.0` controlling how aggressively background audio is suppressed when `voice_focus` is set — higher = more aggressive. | +| `language_codes` | **universal-3-5-pro only.** Optional **list** of ISO 639-1 codes (the connection param is **plural** — there is no singular `language_code` input param). Steers output toward the given languages on a per-token basis while still allowing native code-switching among them. Pass the languages you expect (e.g. `["en", "es"]`), or a single-element list (e.g. `["es"]`) for a monolingual session. When unset, no steering is applied and the model code-switches natively across all its supported languages. Distinct from `language_detection` (which only reports the detected language). Accepted codes: en, es, fr, de, it, pt, tr, nl, sv, no, da, fi, hi, vi, ar, he, ja, zh. | +| `language_detection` | **universal-3-5-pro and universal-streaming-multilingual only.** Boolean (default `false`). When `true`, each `Turn` message includes the detected `language_code` and `language_confidence`. universal-3-5-pro natively code-switches without this — use it only when you need the per-turn language reported. | | `llm_gateway` | JSON-stringified LLM Gateway config — triggers LLM analysis on each completed turn, results delivered as `LLMGatewayResponse` messages | ### Messages Sent (Client to Server) @@ -58,7 +58,7 @@ For new realtime/streaming code, use **`speech_model=universal-3-5-pro`** by def - **Begin:** Session start confirmation, includes session `id` - **Turn:** Transcript data with `transcript` text, `end_of_turn` boolean flag, and `words` array -- **SpeechStarted:** Voice Activity Detection (VAD) event indicating speech has begun (U3 Pro only — use for barge-in detection) +- **SpeechStarted:** Voice Activity Detection (VAD) event indicating speech has begun (universal-3-5-pro only — use for barge-in detection) - **SpeakerRevision:** Revised speaker labels at session close (only when `speaker_labels` is enabled). See Streaming Diarization below. - **LLMGatewayResponse:** LLM analysis result for the completed turn (only present when `llm_gateway` connection parameter is set) - **Termination:** Session end confirmation @@ -88,14 +88,14 @@ Streaming is billed on **WebSocket-open duration per session**, and concurrent s ### universal-3-5-pro (recommended default) - Next-generation flagship streaming model; use it by default in new realtime/streaming integrations -- More languages than u3-rt-pro: EN, ES, DE, FR, PT, IT, TR, NL, SV, NO, DA, FI, HI, VI, AR, HE, JA, ZH -- Improved prompting and enhanced conversational-context features; supports `mode`, `prompt`, `agent_context`, and language detection like u3-rt-pro +- 18 languages with native code-switching: EN, ES, DE, FR, PT, IT, TR, NL, SV, NO, DA, FI, HI, VI, AR, HE, JA, ZH +- Punctuation-based turn detection, promptable, and enhanced conversational-context features; supports `mode`, `prompt`, `keyterms_prompt`, `agent_context`, `language_codes`, and language detection +- The only U3-Pro-family streaming model still in the spec — supersedes the removed `u3-rt-pro` -### u3-rt-pro +### u3-rt-pro (legacy) -- Universal-3 Pro Streaming — use when you intentionally need Universal-3 Pro behavior -- 6 languages (EN, ES, DE, FR, PT, IT) with native code-switching -- Punctuation-based turn detection, promptable, supports the `mode` accuracy/latency tradeoff +- Universal-3 Pro Streaming (6 languages: EN, ES, DE, FR, PT, IT) +- **Removed July 2026** from the streaming docs, model picker, and the `speech_model` spec enum; superseded by `universal-3-5-pro`, which offers more languages and improved prompting/context. New integrations should use `universal-3-5-pro`. ### universal-streaming-english @@ -118,13 +118,13 @@ Streaming is billed on **WebSocket-open duration per session**, and concurrent s ## Turn Detection -### U3 Pro +### Universal-3.5 Pro -Uses **punctuation-based** turn detection (`.` `?` `!`). The `end_of_turn_confidence_threshold` parameter has **NO effect** on U3 Pro models. +Uses **punctuation-based** turn detection (`.` `?` `!`). The `end_of_turn_confidence_threshold` parameter has **NO effect** on universal-3-5-pro. Turn-end timing is set by the `mode` preset and tuned via `min_turn_silence`/`max_turn_silence` (defaults: `max_turn_silence` 1536ms, or 768ms when `speaker_labels` is enabled). ### Universal Streaming -Uses **confidence-based** turn detection. The `end_of_turn_confidence_threshold` defaults to `0.4`. +Uses **confidence-based** turn detection. The `end_of_turn_confidence_threshold` defaults to `0.4` (Universal Streaming English/Multilingual only); `max_turn_silence` defaults to 1280ms. ### Entity Splitting Caveat @@ -137,7 +137,7 @@ A low `min_turn_silence` value can split entities like phone numbers across turn Change settings mid-stream without reconnecting. Fields are model-dependent: - **Universal Streaming:** `keyterms_prompt`, `min_turn_silence`, `max_turn_silence` -- **universal-3-5-pro / u3-rt-pro:** `mode`, `prompt`, `keyterms_prompt`, `min_turn_silence`, `max_turn_silence`, `continuous_partials`, `vad_threshold`, `interruption_delay`, `agent_context` +- **universal-3-5-pro:** `mode`, `prompt`, `keyterms_prompt`, `min_turn_silence`, `max_turn_silence`, `continuous_partials`, `vad_threshold`, `interruption_delay`, `agent_context` Send a JSON message: @@ -199,7 +199,7 @@ Authorization: API_KEY Real-time PII redaction in streaming sessions. Detected PII is replaced in **final turns only** before being sent to the client. -- Supported models: `universal-3-5-pro`, `u3-rt-pro`, `universal-streaming-english`, `universal-streaming-multilingual` +- Supported models: `universal-3-5-pro`, `universal-streaming-english`, `universal-streaming-multilingual` - When `redact_pii=true`, `include_partial_turns` defaults to `false` automatically — partials would otherwise leak unredacted text - Audio redaction is **not** available for streaming. For redacted audio files, use [pre-recorded PII redaction](https://www.assemblyai.com/docs/guardrails/pii-redaction) with `redact_pii_audio` - Same policies as pre-recorded redaction (`person_name`, `phone_number`, `email_address`, `credit_card_number`, `us_social_security_number`, `date_of_birth`, etc.) @@ -234,7 +234,7 @@ Enable speaker diarization by setting query parameters on the WebSocket URL: ### Revised speaker labels (SpeakerRevision) -When the session ends, the server runs a final refinement pass over the whole conversation and emits a **single `SpeakerRevision` message** (when `speaker_labels` is enabled). It arrives **right before `Termination`**, after the client sends `Terminate`. (Streaming diarization itself is supported across current streaming models; the `SpeakerRevision` message is defined in the Universal-3 Pro-family streaming message set.) +When the session ends, the server runs a final refinement pass over the whole conversation and emits a **single `SpeakerRevision` message** (when `speaker_labels` is enabled). It arrives **right before `Termination`**, after the client sends `Terminate`. (Streaming diarization itself is supported across current streaming models; the `SpeakerRevision` message is defined for universal-3-5-pro.) - A session emits **zero or one** `SpeakerRevision` message. - It contains a `revisions` array with **only the turns whose speaker labels changed** — unchanged turns are omitted. @@ -261,11 +261,11 @@ When the session ends, the server runs a final refinement pass over the whole co --- -## Context Carryover (universal-3-5-pro / u3-rt-pro) +## Context Carryover (universal-3-5-pro) -Universal-3 Pro-family streaming models automatically carry prior **finalized** turns (`end_of_turn: true`) forward as context to improve accuracy on the next turn. This is **on by default** — no configuration required — and is per-session (closing the WebSocket clears it). +universal-3-5-pro automatically carries prior **finalized** turns (`end_of_turn: true`) forward as context to improve accuracy on the next turn. This is **on by default** — no configuration required — and is per-session (closing the WebSocket clears it). -**Defaults:** context carryover enabled, ~3 prior entries carried (controlled by `previous_context_n_turns`, default `3`), ~1500-character max context. Older entries drop first. Set `previous_context_n_turns: 0` at connection time to disable automatic context carryover entirely. +**Defaults:** context carryover enabled, up to **5** prior entries carried (controlled by `previous_context_n_turns`, server default `5`, range 0–100). Older entries drop first. Set `previous_context_n_turns: 0` at connection time to disable automatic context carryover entirely. You can additionally pass your voice agent's spoken reply (TTS text) via **`agent_context`** so the model knows the question the user is about to answer — especially valuable for short replies (`"yes"`, `"7pm"`, a single name) and spelled-out entities (emails, account IDs). For example, after the agent asks `"What's your email address?"`, `agent_context` helps the model produce `"user@assemblyai.com"` instead of `"user at assemblyai dot com"`. @@ -278,11 +278,11 @@ Two ways to set it: { "type": "UpdateConfiguration", "agent_context": "Sure — what date would you like to book?" } ``` -**Limits:** `universal-3-5-pro` or `u3-rt-pro` only. Per-value cap ~1500 chars. Not billed separately (streaming is billed on session duration). +**Limits:** `universal-3-5-pro` only. Per-value cap 1750 chars (`agent_context` and `prompt`). Not billed separately (streaming is billed on session duration). --- -## Voice Focus (Noise Suppression, universal-3-5-pro / u3-rt-pro) +## Voice Focus (Noise Suppression, universal-3-5-pro) Voice Focus isolates the primary voice and suppresses background chatter, keyboard clicks, fan hum, and room echo **before** the audio reaches the transcription model. Set the `voice_focus` connection parameter when you open the WebSocket. Pick the variant by how close the speaker is to the mic: @@ -291,7 +291,7 @@ Voice Focus isolates the primary voice and suppresses background chatter, keyboa | Near field | `near-field` | Headsets, handsets, and other close-talking microphones | | Far field | `far-field` | Conference rooms, drive-thru speakers, laptop mics, other distant capture | -Optionally tune `voice_focus_threshold` (float `0.0`–`1.0`) to control how aggressively background audio is suppressed — higher = more aggressive. Omit `voice_focus` to disable. Universal-3 Pro-family streaming models only. +Optionally tune `voice_focus_threshold` (float `0.0`–`1.0`, default `0.7`) to control how aggressively background audio is suppressed — higher = more aggressive. Omit `voice_focus` to disable. universal-3-5-pro only. ```python CONNECTION_PARAMS = { diff --git a/skills/assemblyai/references/voice-agents.md b/skills/assemblyai/references/voice-agents.md index 9d17ccc..c160069 100644 --- a/skills/assemblyai/references/voice-agents.md +++ b/skills/assemblyai/references/voice-agents.md @@ -3,8 +3,8 @@ AssemblyAI supports four paths for building voice agents: 1. **Speech-to-Speech API** — single WebSocket for full voice agent (speech-in → LLM → speech-out) -2. **LiveKit Agents** — fastest path to deployment using U3 Pro STT -3. **Pipecat (by Daily)** — open-source, maximum customizability using U3 Pro STT +2. **LiveKit Agents** — fastest path to deployment using Universal-3.5 Pro STT +3. **Pipecat (by Daily)** — open-source, maximum customizability using Universal-3.5 Pro STT 4. **Direct WebSocket** — fully custom STT builds (see `streaming.md`) ## Voice Agent API @@ -125,6 +125,10 @@ The first `session.update` configures the agent one of **two mutually exclusive* `agent_id` is **mutually exclusive** with the inline fields — sending both in the same `session.update` raises a validation error. `agent_id` can only be set in the **first** `session.update`, before `session.ready`. After binding, you can still send later `session.update` messages to change mutable fields. +### Bring your own LLM (`llm`) + +By default a voice agent uses AssemblyAI's **managed** conversational model. To run it on a different model, set the `llm` field on a **stored agent** (via the Agents REST API — this is a REST/create-time field, *not* a `session.update` field) to your own **OpenAI-compatible** endpoint: `{"llm": [{"base_url": "https://api.openai.com/v1", "model": "gpt-4o-mini", "api_key": "sk-..."}]}`. The list accepts **one entry today**; the model must support **streamed** chat completions; `api_key` is write-only. Send `"llm": []` to revert to the managed model. To use a frontier model without your own provider account, point `base_url` at the LLM Gateway (`https://llm-gateway.assemblyai.com/v1`) and pass your AssemblyAI key. Latency then depends on your endpoint. See `references/api-reference.md` for the full schema. + ### Tool Definition Fields | Field | Type | Default | Notes | @@ -243,7 +247,7 @@ For browser apps, enable echo cancellation via `getUserMedia({ audio: { echoCanc Set a voice via `session.output.voice` in `session.update` **before `session.ready`**. `output.voice` and `output.format` are immutable once the session is established — the voice **cannot be changed mid-conversation**. (`output.volume` is the exception — it remains mutable.) Default is `ivy`. -**Every voice speaks all output languages** — 🇺🇸 English, French, German, Italian, Portuguese, Spanish, Hindi, Mandarin, Russian, Korean, and Japanese. The two groups below differ only by the voice's *primary accent*, not which languages it can speak. (Input recognition covers en/fr/de/it/pt/es only, so an agent can speak an output language it can't transcribe — useful for translation-style flows.) +**Every voice speaks all output languages** — 🇺🇸 English, French, German, Italian, Portuguese, Spanish, Hindi, Mandarin, Russian, Korean, and Japanese. The two groups below differ only by the voice's *primary accent*, not which languages it can speak. Officially supported **output** languages (those with at least one native/primary-accent voice) are English, French, Italian, Spanish, Hindi, and Russian; German, Portuguese, Turkish, Dutch, and Swedish are on the roadmap. **Input** recognition uses Universal-3.5 Pro Streaming and covers **18 languages** with native code-switching (en, es, de, fr, pt, it, tr, nl, sv, no, da, fi, hi, vi, ar, he, ja, zh) — an agent can speak an output language it can't transcribe (useful for translation-style flows). **American-English accent** (carried into other languages): `ivy`, `james`, `tyler`, `winter`, `bella`, `david`, `kyle`, `helen`, `martha`, `river`, `emma`, `victor`, `eleanor` @@ -286,19 +290,21 @@ In browsers, pre-handshake failures (like `UNAUTHORIZED`) surface as `close` eve **`universal-3-5-pro`** is the recommended default model for new STT-based voice agent work. The raw streaming API also defaults to it when `speech_model` is omitted; set it explicitly when pinning behavior or using an SDK that requires the field. -| Feature | universal-3-5-pro | u3-rt-pro | universal-streaming-english | universal-streaming-multilingual | -|---------|-------------------|-----------|------------------------------|----------------------------------| -| Turn detection | Punctuation-based | Punctuation-based | Confidence-based | Confidence-based | -| Custom prompting | Yes | Yes | No | No | -| Keyterms boosting | Yes | Yes | Yes | Yes | -| Speaker diarization | Yes | Yes | Yes | Yes | -| Dynamic mid-session updates | Yes | Yes | Yes | Yes | -| Multilingual code switching | Yes | Yes | No | Yes | -| Languages | 18 (en, es, de, fr, pt, it, tr, nl, sv, no, da, fi, hi, vi, ar, he, ja, zh) | 6 (en, es, fr, de, it, pt) | English only | Multiple | +| Feature | universal-3-5-pro | universal-streaming-english | universal-streaming-multilingual | +|---------|-------------------|------------------------------|----------------------------------| +| Turn detection | Punctuation-based | Confidence-based | Confidence-based | +| Custom prompting | Yes | No | No | +| Keyterms boosting | Yes | Yes | Yes | +| Speaker diarization | Yes | Yes | Yes | +| Dynamic mid-session updates | Yes | Yes | Yes | +| Multilingual code switching | Yes | No | Yes | +| Languages | 18 (en, es, de, fr, pt, it, tr, nl, sv, no, da, fi, hi, vi, ar, he, ja, zh) | English only | Multiple | + +(`u3-rt-pro` / Universal-3 Pro Streaming was removed from the streaming model picker and spec enum in July 2026 — superseded by `universal-3-5-pro`.) -`end_of_turn_confidence_threshold` does NOT work with `universal-3-5-pro` or `u3-rt-pro` — it only applies to older universal-streaming models. +`end_of_turn_confidence_threshold` does NOT work with `universal-3-5-pro` — it only applies to the Universal Streaming (English/Multilingual) models. -## Turn Detection (universal-3-5-pro / u3-rt-pro) +## Turn Detection (universal-3-5-pro) 1. User pauses for `min_turn_silence` (e.g., 100ms) 2. Model checks for terminal punctuation (`.` `?` `!`) @@ -308,7 +314,7 @@ In browsers, pre-handshake failures (like `UNAUTHORIZED`) surface as `close` eve ## Silence Settings by Use Case -These are for **Universal Streaming** models. U3 Pro-family defaults differ. +These are for **Universal Streaming** models. universal-3-5-pro defaults differ (`max_turn_silence` 1536ms, or 768ms with `speaker_labels`). | Profile | min_turn_silence | max_turn_silence | Use Case | |---------|-----------------|-----------------|----------|