AssemblyAI · dlange-aai · Jul 6, 2026 · Jul 6, 2026
diff --git a/skills/assemblyai/SKILL.md b/skills/assemblyai/SKILL.md
diff --git a/skills/assemblyai/references/api-reference.md b/skills/assemblyai/references/api-reference.md
@@ -32,7 +32,7 @@ Submit an audio file for transcription. Send a JSON body with the parameters bel
 | Parameter | Type | Description |
 |---|---|---|
 | `audio_url` | string | **Required.** URL of the audio file to transcribe. Can be a public URL or an `upload_url` from the upload endpoint. |
-| `speech_models` | array | **Optional** (as of June 2026). Priority-ordered list of speech models to use (e.g., `["universal-3-5-pro", "universal-2"]`). First model is used if supported; falls back to next. If omitted, defaults to `["universal-3-pro", "universal-2"]`. Universal-3.5 Pro is accepted here (`["universal-3-5-pro"]`). |
+| `speech_models` | array | **Optional.** Priority-ordered list of speech models. First supported model is used; falls back to the next. **If omitted, defaults to `["universal-3-5-pro", "universal-2"]`.** The enum now accepts only `universal-3-5-pro` and `universal-2` — `universal-3-pro` was removed (superseded by `universal-3-5-pro`). The `speech_model_used` response field reports which model actually ran. |
 | `prompt` | string | For Universal-3.5 Pro, a contextual *description* of the audio (domain → scenario → full detail), **not** formatting/behavioral instructions (those are ignored). **Complementary with `keyterms_prompt`** — both can be set together. |
 | `keyterms_prompt` | array | List of key terms/phrases (strings) to boost recognition accuracy — up to **1000** terms for Universal-3.5 Pro, **200** for Universal-2, max **6 words per phrase**. **Complementary with `prompt`** — both can be set together. |
 | `language_code` | string | Language code (e.g., `"en_us"`, `"es"`, `"fr"`). Defaults to `"en_us"`. |
@@ -42,11 +42,11 @@ Submit an audio file for transcription. Send a JSON body with the parameters bel
 | `speaker_labels` | boolean | Enable speaker diarization. Default `false`. |
 | `sentiment_analysis` | boolean | Enable sentiment analysis on each sentence. Default `false`. |
 | `entity_detection` | boolean | Enable entity detection. Default `false`. |
-| `auto_chapters` | boolean | **Deprecated.** Enable auto chapters. Use LLM Gateway instead. |
+| `auto_chapters` | boolean | **Deprecated.** Use Speech Understanding `summarization` (chaptered summary; see `speech-understanding.md`) or the LLM Gateway instead. |
 | `iab_categories` | boolean | Enable IAB content category detection. Default `false`. |
 | `content_safety` | boolean | Enable content safety detection. Default `false`. |
 | `content_safety_confidence` | integer | Minimum confidence threshold (25-100) for content safety labels. |
-| `summarization` | boolean | **Deprecated.** Enable summarization. Use LLM Gateway instead. |
+| `summarization` | boolean | **Deprecated.** Use Speech Understanding `summarization`/`action_items` (see `speech-understanding.md`) or the LLM Gateway instead. |
 | `summary_model` | string | **Deprecated.** Model for summarization. |
 | `summary_type` | string | **Deprecated.** Type of summary. |
 | `redact_pii` | boolean | Enable PII redaction. Default `false`. |
@@ -68,10 +68,10 @@ Submit an audio file for transcription. Send a JSON body with the parameters bel
 | `speech_understanding` | object | Enable Speech Understanding inline. Features nest under `speech_understanding.request` (the `request` wrapper is required): `translation`, `speaker_identification`, and/or `custom_formatting`. See `speech-understanding.md`. |
 | `speakers_expected` | integer | Hint for number of speakers (diarization). Deprecated in favor of `speaker_options`. |
 | `speaker_options` | object | Diarization options: `min_speakers_expected` (int, default 1), `max_speakers_expected` (int). |
-| `temperature` | float | 0–1. Controls randomness. Universal-3 Pro only. |
-| `domain` | string | Domain-specific model variant. `"medical-v1"` enables Medical Mode (EN, ES, DE, FR). Supported on Universal-3 Pro and Universal-2. |
-| `remove_audio_tags` | string | Remove inline annotations from the transcript. `"all"` removes all (audio event markers and speaker cues); `"speaker"` removes only speaker cues while keeping other annotations. Universal-3 Pro only. |
-| `language_codes` | array | List of language codes for code-switching (must include `"en"`). Universal-3 Pro only. |
+| `temperature` | float | 0–1. Controls randomness. Universal-3.5 Pro only. |
+| `domain` | string | Domain-specific model variant. `"medical-v1"` enables Medical Mode (EN, ES, DE, FR). Supported on Universal-3.5 Pro and Universal-2. |
+| `remove_audio_tags` | string | Remove inline annotations from the transcript. `"all"` removes all (audio event markers and speaker cues); `"speaker"` removes only speaker cues while keeping other annotations. Universal-3.5 Pro only. |
+| `language_codes` | array | List of language codes for code-switching (must include `"en"`). Universal-3.5 Pro only. |
 | `audio_start_from` | integer | Start transcription from this time offset, in **milliseconds**. |
 | `audio_end_at` | integer | End transcription at this time offset, in **milliseconds**. |
 | `speech_threshold` | float | Confidence threshold (0-1) for filtering low-confidence speech. Requires at least **30 seconds** of audio. |
@@ -105,7 +105,7 @@ The transcript response may include an optional `metadata` object with additiona
   "metadata": {
     "domain_used": null,
     "warnings": [
-      { "message": "'ja' is not supported in universal-3-pro — transcription is handled by universal-2. To silence this warning, set speech_models: [\"universal-3-pro\", \"universal-2\"]." }
+      { "message": "'ur' is not supported in universal-3-5-pro — transcription is handled by universal-2. To silence this warning, set speech_models: [\"universal-3-5-pro\", \"universal-2\"]." }
     ]
   }
 }
@@ -479,15 +479,55 @@ A REST API for creating **reusable** voice agents. An agent stores its `system_p
 |---------------|-------------|
 | `POST /v1/agents` | Create an agent. Returns `201` with the full record including a generated `id`. |
 | `GET /v1/agents` | List agents (lightweight records, no tools/prompt), newest first. |
-| `GET /v1/agents/{agent_id}` | Retrieve one agent. Tool header **values** are masked as `"***"`. |
+| `GET /v1/agents/{agent_id}` | Retrieve one agent. Read responses return tool-header **names** and `last_set_at` only — header **values are never returned** (write-only, encrypted at rest; they are *not* masked as `"***"`). Likewise a connected LLM (`llm`) comes back as `base_url` + `model` only, never `api_key`. |
 | `PUT /v1/agents/{agent_id}` | Update an agent. Every field optional — send only what you want to change. |
 | `DELETE /v1/agents/{agent_id}` | Delete an agent. Returns `204`, no body. |
 
-**Create body** (`application/json`): required `name`, `system_prompt`, `voice`; optional `greeting`, `input`, `output`, `tools`. Note `voice` is a **top-level** field here (in the WebSocket `session.update` it lives under `output.voice`). `greeting` is spoken straight to TTS on connect — omit it to have the agent listen first.
+**Create body** (`application/json`): required `name`, `system_prompt`, `voice`; optional `greeting`, `input`, `output`, `tools`, `llm`. Note `voice` is a **top-level** field here (in the WebSocket `session.update` it lives under `output.voice`). `greeting` is spoken straight to TTS on connect — omit it to have the agent listen first.
 
 ```bash
 curl -X POST https://agents.assemblyai.com/v1/agents \
   -H 'Authorization: YOUR_API_KEY' \
   -H 'Content-Type: application/json' \
   -d '{"name":"Support Bot","system_prompt":"You are a concise support agent.","voice":"ivy","greeting":"Hi, how can I help?"}'
 ```
+
+### Server-side HTTP tools (`headers` shape)
+
+A tool with an `http` block (`url`, `http_method`, `headers`) is executed **server-side** — AssemblyAI makes the request on your behalf. As of June 2026 `headers` is a **list of `{name, value}` entries**, NOT a `{name: value}` dict:
+
+```json
+{ "http": { "url": "https://api.example.com/orders", "http_method": "POST",
+  "headers": [ { "name": "Authorization", "value": "Bearer xyz" } ] } }
+```
+
+- **Write (create/update):** each entry is `{ name, value?, remove? }`. Provide `value` to set/rotate; provide name only to keep the stored value unchanged (round-tripping on update); `remove: true` deletes it. Sending both `value` and `remove: true` is rejected.
+- **Read (get/list):** each entry is `{ name, last_set_at }` — values are never returned.
+- Server-side constraints: `https` + public hosts only (private/loopback/CGNAT blocked); redirects not followed; response body capped at 8 KiB; per-call timeout from `timeout_seconds`.
+
+### Bring your own LLM (`llm`)
+
+Point an agent at your own **OpenAI-compatible** chat-completions endpoint instead of AssemblyAI's managed model. Set the `llm` field (a list; **only one entry accepted today**) on create/update:
+
+```json
+{ "llm": [ { "base_url": "https://api.openai.com/v1", "model": "gpt-4o-mini", "api_key": "sk-..." } ] }
+```
+
+- `base_url` (required, `https` + public host — the agent calls `POST {base_url}/chat/completions`), `model` (required), `api_key` (required, **write-only**, never returned).
+- Must support **streamed** chat completions. Send `"llm": []` to switch back to the managed model.
+- To use a frontier model without your own provider account, point `base_url` at the **LLM Gateway** (`https://llm-gateway.assemblyai.com/v1`, EU: `https://llm-gateway.eu.assemblyai.com/v1`) and pass your AssemblyAI key as `api_key`.
+
+## 18. Voice Agent Webhooks API
+
+A REST API (base URL `https://agents.assemblyai.com`, same auth) to subscribe URLs to Voice Agent lifecycle events.
+
+| Method & Path | Description |
+|---------------|-------------|
+| `POST /v1/webhook-subscriptions` | Subscribe a URL to one or more events. Scope to one agent with `agent_id`, or omit for account-wide events. Returns the created subscription. |
+| `GET /v1/webhook-subscriptions` | List subscriptions (paginated). |
+| `GET /v1/webhook-subscriptions/{subscription_id}` | Retrieve one subscription. |
+| `PUT /v1/webhook-subscriptions/{subscription_id}` | Update `url`, `events`, `secret`, or `enabled`. Sending a new `secret` rotates it and increments `secret_version`. |
+| `DELETE /v1/webhook-subscriptions/{subscription_id}` | Delete a subscription. Returns `204`. |
+
+- **Events:** `session.started`, `session.completed`, `call.connected`, `call.ended`, `call.failed`.
+- **Body fields:** `url` (`https` + public host, required), `events` (array, required), `secret` (32–256 printable ASCII chars, no whitespace — signing secret, **write-only**, only its `secret_version` is returned), `agent_id` (optional; omit for account-wide).
diff --git a/skills/assemblyai/references/audio-intelligence.md b/skills/assemblyai/references/audio-intelligence.md
@@ -77,7 +77,7 @@ Legacy parameters (still functional but not recommended for new code):
 
 - Enable with `domain: "medical-v1"` in the request body
 - Improves accuracy for medical terminology: medications, procedures, conditions, dosages
-- Supported models: Universal-3 Pro, Universal-2 (pre-recorded); all streaming models
+- Supported models: Universal-3.5 Pro, Universal-2 (pre-recorded); all streaming models
 - Supported languages: English, Spanish, German, French only
 - Billed as a separate add-on
 - If used with an unsupported language, the API ignores `domain` and returns a warning — no charge is applied

diff --git a/skills/assemblyai/references/python-sdk.md b/skills/assemblyai/references/python-sdk.md
@@ -47,7 +47,7 @@ Use `speech_models` to specify a preferred model with automatic fallback:
 
 ```python
 config = aai.TranscriptionConfig(
-    speech_models=["universal-3-pro", "universal-2"]
+    speech_models=["universal-3-5-pro", "universal-2"]
 )
 
 transcriber = aai.Transcriber(config=config)

diff --git a/skills/assemblyai/references/speech-understanding.md b/skills/assemblyai/references/speech-understanding.md
@@ -2,7 +2,9 @@
 
 ## Overview
 
-Speech Understanding provides post-transcription intelligence: **Translation, Speaker Identification, and Custom Formatting**. (Entity Detection, Sentiment Analysis, Key Phrases, and Topic Detection are *not* part of this object — they remain classic boolean transcript params; see `audio-intelligence.md`.)
+Speech Understanding provides post-transcription intelligence: **Translation, Speaker Identification, Custom Formatting, Summarization, and Action Items**. (Entity Detection, Sentiment Analysis, Key Phrases, and Topic Detection are *not* part of this object — they remain classic boolean transcript params; see `audio-intelligence.md`.)
+
+> **Summarization vs the deprecated `summarization`/`auto_chapters` params:** the top-level `summarization` and `auto_chapters` boolean params on `POST /v2/transcript` are **deprecated**. The modern replacements are the Speech Understanding `summarization` (chaptered summary with timestamps + headlines) and `action_items` features below — or the LLM Gateway for fully custom prompts.
 
 Two ways to run it:
 
@@ -29,7 +31,7 @@ Auth header: `Authorization: API_KEY` (no `Bearer` prefix).
 
 Translates the transcript into one or more target languages.
 
-- **Models:** Universal-3 Pro and Universal-2 only. **Regions:** US & EU. Supports 100+ language codes.
+- **Models:** Universal-3.5 Pro and Universal-2 only. **Regions:** US & EU. Supports 100+ language codes.
 
 ### Parameters (`speech_understanding.request.translation`)
 
@@ -181,6 +183,93 @@ The format params are **strings (format patterns), not booleans**:
 
 `speech_understanding.response.custom_formatting` contains `formatted_text`, `formatted_utterances` (only when `format_utterances: true`), and a `mapping` object (original → formatted).
 
+## Summarization
+
+Generates a chaptered summary of the transcript — each chapter has timestamps, a `text` summary, and a `headline`. Replaces the deprecated top-level `summarization`/`auto_chapters` params. **Open beta.** Models: `universal-3-5-pro`, `universal-2`. Regions: US & EU.
+
+### Parameters (`speech_understanding.request.summarization`)
+
+- `summary_type` (string, **required**): `"paragraph"` (longer, more detailed) or `"bullets"` (short, concise).
+- `effort` (string, default `"low"`): `"low"` or `"medium"`. `medium` spends more processing power for higher-quality summaries — use it for high-stakes meetings, multilingual audio, or very long audio (≈1.5h+). Default `low` is sufficient for most cases.
+
+### Example
+
+```json
+{
+  "audio_url": "https://example.com/audio.mp3",
+  "speech_understanding": {
+    "request": {
+      "summarization": { "summary_type": "paragraph", "effort": "low" }
+    }
+  }
+}
+```
+
+### Response
+
+`speech_understanding.response.summarization` → `{ status, summary_type, effort, summary: [{ start, end, text, headline }] }`:
+
+```json
+{
+  "speech_understanding": {
+    "response": {
+      "summarization": {
+        "status": "success",
+        "summary_type": "paragraph",
+        "effort": "low",
+        "summary": [
+          { "start": 240, "end": 37100, "text": "Smoke from hundreds of Canadian wildfires...", "headline": "Smoke from Canadian Wildfires Affects US Air Quality" }
+        ]
+      }
+    }
+  }
+}
+```
+
+## Action Items
+
+Extracts action items from the transcript, each with the source `quote` and a `timestamp`. **Open beta.** Models: `universal-3-5-pro`, `universal-2`. Regions: US & EU.
+
+### Parameters (`speech_understanding.request.action_items`)
+
+- `include_decisions` (boolean, default `false`): when `true`, also captures decisions made in the conversation as action items.
+- `effort` (string, default `"low"`): `"low"` or `"medium"` (see Summarization for guidance on `medium`).
+
+Pass an empty object (`"action_items": {}`) to use defaults.
+
+### Example
+
+```json
+{
+  "audio_url": "https://example.com/audio.mp3",
+  "speech_understanding": {
+    "request": {
+      "action_items": { "include_decisions": true }
+    }
+  }
+}
+```
+
+### Response
+
+`speech_understanding.response.action_items` → `{ status, effort, items: [{ action_item, quote, timestamp }] }`:
+
+```json
+{
+  "speech_understanding": {
+    "response": {
+      "action_items": {
+        "status": "success",
+        "effort": "low",
+        "items": [
+          { "action_item": "Observe the expansion and contraction of the Arctic ice cap throughout the seasons.", "quote": "The Arctic ice cap ... expands in winter and contracts in summer.", "timestamp": 9520 }
+        ]
+      }
+    }
+  }
+}
+```
+
 ## Post-hoc via the Understanding Endpoint
 
 To run Speech Understanding on an **existing** transcript, POST to `/v1/understanding` with a `transcript_id` (same `speech_understanding.request.<feature>` structure):