From 2fdd5bf4c1beda2ddb97b99737fdf911513cc256 Mon Sep 17 00:00:00 2001 From: David Lange Date: Fri, 10 Jul 2026 12:08:07 -0400 Subject: [PATCH] Update AssemblyAI skill: remove gemini-3-flash-preview, add spoken-digit tool pattern trap + granular location PII policies - LLM Gateway: drop retired gemini-3-flash-preview model (removed from spec #198) - Voice Agents: document tool parameter pattern/examples hints for spoken digit sequences and the re-ask-loop trap from over-strict patterns (#208) - API reference: explain shape-dependent matching of granular location_* PII redaction policies; Entity Detection only returns broad location (#201) - SKILL.md: add Key Gotcha for spoken-digit tool pattern trap --- skills/assemblyai/SKILL.md | 1 + skills/assemblyai/references/api-reference.md | 10 ++++++++++ skills/assemblyai/references/llm-gateway.md | 1 - skills/assemblyai/references/voice-agents.md | 19 +++++++++++++++++++ 4 files changed, 30 insertions(+), 1 deletion(-) diff --git a/skills/assemblyai/SKILL.md b/skills/assemblyai/SKILL.md index 63306e2..fd22803 100644 --- a/skills/assemblyai/SKILL.md +++ b/skills/assemblyai/SKILL.md @@ -161,6 +161,7 @@ See `references/llm-gateway.md` for models, tool calling, structured outputs, an | Voice Agent HTTP tool `headers` | Server-side HTTP tools take `headers` as a **list of `{name, value}` entries** (since June 2026), NOT a `{name: value}` dict. Reads return `{name, last_set_at}` only — values are write-only and never returned (not masked as `"***"`) | | Voice Agent BYO LLM (`llm`) | Set `llm` on a **stored agent** (REST create/update, not `session.update`) to run on your own OpenAI-compatible endpoint: `[{base_url, model, api_key}]` — one entry only, must stream, `api_key` write-only. `"llm": []` reverts to the managed model. Point `base_url` at the LLM Gateway to use a frontier model on your AssemblyAI account | | Voice Agent webhooks | Subscribe to Voice Agent events via `POST https://agents.assemblyai.com/v1/webhook-subscriptions` (events: `session.started`/`session.completed`/`call.connected`/`call.ended`/`call.failed`; scope with `agent_id` or omit for account-wide). See `references/api-reference.md` | +| Voice Agent tool `pattern` on spoken digits | A tool parameter's `pattern` is matched against the value **as the agent produces it from speech**, and cleanup is best-effort. Long digit sequences (card/account numbers) often arrive with interior spaces (`"4 2 4 2 …"`), so a strict `^\d{16}$` rejects them and traps the caller in a re-ask loop. Allow interior spaces, bound the **digit** count, add a spaced `example`, and strip non-digits in your handler. See `references/voice-agents.md` | ## Common Mistakes diff --git a/skills/assemblyai/references/api-reference.md b/skills/assemblyai/references/api-reference.md index b928ae3..3d95986 100644 --- a/skills/assemblyai/references/api-reference.md +++ b/skills/assemblyai/references/api-reference.md @@ -286,6 +286,16 @@ Full list of supported PII policy values for `redact_pii_policies`: `account_number`, `banking_information`, `blood_type`, `credit_card_cvv`, `credit_card_expiration`, `credit_card_number`, `date`, `date_interval`, `date_of_birth`, `drivers_license`, `drug`, `duration`, `email_address`, `event`, `filename`, `gender`, `gender_sexuality`, `healthcare_number`, `injury`, `ip_address`, `language`, `location`, `location_address`, `location_address_street`, `location_city`, `location_coordinate`, `location_country`, `location_state`, `location_zip`, `marital_status`, `medical_condition`, `medical_process`, `money_amount`, `nationality`, `number_sequence`, `occupation`, `organization`, `organization_medical_facility`, `passport_number`, `password`, `person_age`, `person_name`, `phone_number`, `physical_attribute`, `political_affiliation`, `religion`, `sexuality`, `statistics`, `time`, `url`, `us_social_security_number`, `username`, `vehicle_id`, `zodiac_sign` +### Granular `location_*` policies (matching is shape-dependent) + +The `location_*` subtypes let you redact specific parts of a location while leaving others visible — but **which policy matches a span depends on the span's shape**, which is a common source of "why wasn't this redacted" mistakes: + +- A **full contiguous mailing address** (street + city + state + ZIP together, e.g. `145 Windsor St., Toronto, ON M5A 2P5`) is one `location_address` span — redact it with `location_address`. +- A **standalone fragment** is tagged as its specific subtype: a lone street → `location_address_street`, a lone city → `location_city`, a lone state → `location_state`, a lone country → `location_country`, a lone ZIP → `location_zip`, coordinates → `location_coordinate`. +- A **combined phrase that mixes parts without forming a full address** — e.g. `Toronto, Canada` — is a single generic `location` span. The granular subtypes alone will **not** match it; you must include the broad `location` policy. (So adding only `location_city` expecting `Toronto, Canada` to be redacted will silently fail.) + +These subtypes apply to **PII redaction only**. [Entity Detection](https://www.assemblyai.com/docs/speech-understanding/entity-detection) returns every location under the single `location` entity type — never as `location_city`, `location_zip`, etc. + --- ## 14. Data Retention & Compliance diff --git a/skills/assemblyai/references/llm-gateway.md b/skills/assemblyai/references/llm-gateway.md index be90a2a..dccebb4 100644 --- a/skills/assemblyai/references/llm-gateway.md +++ b/skills/assemblyai/references/llm-gateway.md @@ -61,7 +61,6 @@ Model IDs have NO provider prefix (e.g., use `claude-sonnet-4-5-20250929`, not ` | Model | ID | |-------|----| | Gemini 3.5 Flash | `gemini-3.5-flash` | -| Gemini 3 Flash Preview | `gemini-3-flash-preview` | | Gemini 3.1 Flash Lite Preview ⚠️ US-only | `gemini-3.1-flash-lite-preview` | | Gemini 2.5 Pro | `gemini-2.5-pro` | | Gemini 2.5 Flash | `gemini-2.5-flash` | diff --git a/skills/assemblyai/references/voice-agents.md b/skills/assemblyai/references/voice-agents.md index c160069..0364a79 100644 --- a/skills/assemblyai/references/voice-agents.md +++ b/skills/assemblyai/references/voice-agents.md @@ -142,6 +142,25 @@ By default a voice agent uses AssemblyAI's **managed** conversational model. To `session.tools` updates **replace** the previous array (not merge). Pattern: progressive tool reveal — start with minimal tools, add the next phase's tools after each successful `tool.result`. +### Parameter hints for spoken input (`pattern` / `examples`) + +A parameter's JSON Schema `pattern` (Python regex, matched against the **whole** value — no `^`/`$` needed; escape as `\\d` in JSON) and `examples` describe the value **the agent produces from speech**, not the words the caller says. The agent's cleanup of spoken input is **best-effort, not guaranteed**: it is reliable for well-known shapes (phone numbers, emails, dates), but **long digit sequences** (card numbers, account numbers, claim IDs) often arrive **as spoken** — with interior spaces when the caller reads digits one at a time (`"4 2 4 2 …"`, `"4242 4242 4242 4242"`), or with digits missing if recognition dropped some. + +**The trap (#1 cause of re-ask loops):** a `pattern` written only for the tidy form (`^\\d{16}$`) rejects the spaced form, so the agent silently re-asks — the caller just hears "I didn't catch that" on repeat. Instead, allow interior spaces, bound the **digit** count (not character count), and include a spaced `example`, then strip non-digits in your handler: + +```json +{ + "card_number": { + "type": "string", + "description": "The card number as spoken. May contain spaces between digits; 13–19 digits once spaces are removed.", + "pattern": " *([0-9] *){13,19}", + "examples": ["4242424242424242", "4242 4242 4242 4242", "4 2 4 2 4 2 4 2 4 2 4 2 4 2 4 2"] + } +} +``` + +This tolerates a **complete** number spoken with spaces; it should not (and does not) accept a genuinely incomplete one — a partial capture is correctly rejected and re-asked. Keep the `pattern` as a shape hint that admits real spoken input; do length/checksum validation in your handler. + ### Mutability After `session.ready` | Field | Mutable? |