diff --git a/api-reference/asyncapi-asr-realtime.yml b/api-reference/asyncapi-asr-realtime.yml index a35c61e..e7832db 100644 --- a/api-reference/asyncapi-asr-realtime.yml +++ b/api-reference/asyncapi-asr-realtime.yml @@ -106,7 +106,7 @@ channels: Every message in both directions is one MessagePack map in a binary frame, with a `type` field. Send `config` first, wait for `session_started`, then stream `audio`. Ignore fields and message types - you do not know: version 1 only adds them. + you do not know in server messages: version 1 only adds them. operations: receiveAsrClientMessages: @@ -196,8 +196,13 @@ components: description: | The first message on the connection. Send it within 10 seconds of connecting, then wait for `session_started` before sending audio. - An invalid `config` is answered with an `error` (`invalid_config` or - `unsupported_audio`) and close code 4400. + An invalid `config`, including one with an unknown field, is + answered with an `error` (`invalid_config` or `unsupported_audio`) + and close code 4400. A first message that is not a readable + `config` (for example a text frame or JSON) gets `invalid_config` + and close code 4400 too. Send only the documented fields: not + every other field is refused, so check `session_started.config`, + which shows the settings in effect. examples: - name: Voice agent payload: @@ -240,7 +245,13 @@ components: data: type: string format: binary - description: PCM bytes in the configured format, as MessagePack `bin`. Any length; an odd trailing byte waits for the next message. + description: | + The raw PCM bytes in the configured format, as MessagePack + `bin`. Do not base64-encode the audio, and do not send it as a + string or an array of numbers. If you do, you receive a + recoverable `invalid_request`, and those messages are dropped: + nothing in them is transcribed or counted as received audio. + Any length; an odd trailing byte waits for the next message. description: | Audio in the format from `config`. Send it after `session_started`; 80–100 ms per message is a good size. A message can be at most 1 MiB. @@ -298,15 +309,22 @@ components: maxLength: 32 description: Replaces the list of terms. description: | - Only `terms` is accepted here. `history` is ignored in - `configure`; send earlier lines with `context_add`. + Only `terms` is accepted here. A `configure` with `history` + gets `configured` with `ok: false`; send earlier lines with + `context_add`. description: | Changes settings during the session. Answered by `configured`. Every field is optional, and a field you leave out keeps its current value. The one exception: switching `mode` without `silence_timeout_ms` resets it to the new mode's default. If any value is invalid, - nothing in the message is applied. When `mode` changes to or from - `auto`, `configured` can take a few seconds. + nothing in the message is applied. `audio` is fixed for the + session: a `configure` with an unknown field, with an `audio` that + differs from the session's, or with `context.history` gets + `configured` with `ok: false`, and nothing in it is applied. An + `audio` identical to the session's is accepted. As with `config`, + send only the documented fields; `configured.config` shows the + settings in effect. When `mode` changes to or from `auto`, + `configured` can take a few seconds. examples: - payload: type: configure @@ -370,7 +388,9 @@ components: description: Message type identifier description: | Keeps an idle session open. A session closes after 30 seconds without - any client message. + any client message. A WebSocket ping counts as client activity too, + so a client library that sends pings on its own keeps the session + open. examples: - payload: type: keepalive @@ -748,7 +768,7 @@ components: type: integer text: type: string - description: The final transcript of the turn. Can contain speaker tags such as `<|speaker:0|>`. Can be empty. + description: The final transcript of the turn. Can contain speaker tags such as `<|speaker:0|>` and sound tags such as `[laughter]`. Can be empty. words: type: array items: @@ -1013,8 +1033,10 @@ components: description: | An error. When `recoverable` is `false`, the server closes the connection (close codes 4400, 4408, 4413, 4500 or 1009). - `server_restarting` is recoverable but still closes with 1012: - reconnect for a new session. + `idle_timeout` (close 4408) comes after 30 seconds without any + client message, or when no `config` arrives within 10 seconds of + connecting. `server_restarting` is recoverable but still closes + with 1012: reconnect for a new session. examples: - payload: type: error diff --git a/api-reference/endpoint/websocket/asr-realtime.mdx b/api-reference/endpoint/websocket/asr-realtime.mdx index ae0a21a..e5b6260 100644 --- a/api-reference/endpoint/websocket/asr-realtime.mdx +++ b/api-reference/endpoint/websocket/asr-realtime.mdx @@ -37,27 +37,27 @@ model: transcribe-1-pro-realtime | Header | Required | Description | | --------------- | -------- | -------------------------------------------------------------------------------------------------- | -| `Authorization` | Yes | `Bearer `. | +| `Authorization` | Yes | `Bearer `. Write the scheme exactly as `Bearer`, followed by a space and the key. | | `model` | No | `transcribe-1-pro-realtime`, the default and the only realtime model. Any other value returns 400. | -Each open connection holds one of your team's concurrent session slots until it closes. A session cannot be resumed: a new connection is a new session. +Each open connection counts against the same concurrency limit as your other API requests, from the moment it opens until it closes, even while you send no audio. See [Concurrent request limits](/developer-guide/models-pricing/pricing-and-rate-limits#concurrent-request-limits). A session cannot be resumed: a new connection is a new session. If the server refuses the upgrade, it answers with a plain HTTP response whose JSON body is `{"status": , "message": }`: -| Status | Meaning | What to do | -| ------ | --------------------------------------------------------------------- | --------------------------------------------------------- | -| 400 | The `model` header names another model. | Send `transcribe-1-pro-realtime` or leave the header out. | -| 401 | The API key is missing or invalid. | Check the key. | -| 402 | Insufficient API credit. | Top up your balance. | -| 403 | Realtime speech to text is not enabled for your team. | Contact support (see the note above). | -| 429 | Your team is at its concurrency limit, or the service is at capacity. | Retry with backoff. Honor `Retry-After` when present. | -| 503 | The service is restarting. | Retry. | +| Status | Meaning | What to do | +| ------ | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------- | +| 400 | The `model` header names another model. | Send `transcribe-1-pro-realtime` or leave the header out. | +| 401 | The API key is missing or invalid, or the `Authorization` header does not start with `Bearer `. | Check the key and the header. | +| 402 | Insufficient API credit. | Top up your balance. | +| 403 | Realtime speech to text is not enabled for your team. | Contact support (see the note above). | +| 429 | Your team is at its concurrency limit, or the service is at capacity. | Retry with backoff. Honor `Retry-After` when present. | +| 503 | The service is restarting. | Retry. | The `101 Switching Protocols` response carries an `x-request-id` header, and so do the 400 (model), capacity 429 and 503 refusals. It is the same value as `request_id` in `session_started`, `error` and `session_done`. Log it, and include it when you contact support. Refusals for the key, balance, access and your concurrency limit (401, 402, 403 and concurrency 429) happen before a session is assigned and carry no `x-request-id`. ## Framing -- Every message is **one binary WebSocket frame holding one MessagePack map** with a `type` field. Audio bytes are MessagePack `bin`. Do not send text frames. +- Every message is **one binary WebSocket frame holding one MessagePack map** with a `type` field. Audio bytes are MessagePack `bin`. Do not send text frames. JSON is not accepted, in text frames or in binary frames. - Every server message carries `seq`, which increases by one per message within the session. - Times are `*_ms` integers: milliseconds of session audio since the first audio byte, in your audio's own time at the `sample_rate` you configured. - WebSocket ping and pong frames are allowed at any time, including before `config`. The server answers pings, and a ping counts as client activity for the 30-second idle limit. Pings do not extend the 10-second deadline for the first `config`. @@ -91,6 +91,8 @@ In `auto` mode, `draft` messages can come before and after `user_turn_taken`, an The first message on the connection. Send it within 10 seconds of connecting, then wait for `session_started` before you send audio. +Audio sent before `session_started` is not lost: once the session has started, the server reads it in order, and it counts as received audio like any other. If the `config` is refused, none of it is processed. Waiting for `session_started` is still the reliable way to know that your `config` was accepted, and to receive its error and close code if it was not. + ```json { "type": "config", @@ -124,7 +126,7 @@ The first message on the connection. Send it within 10 seconds of connecting, th | `context.terms` | No | `[]` | Up to 100 terms of up to 32 characters each: names, products and jargon to recognize. | | `client_reference_id` | No | | Your own ID for the session, up to 256 characters. It is recorded with the session and not sent back. | -A value of the wrong type for `ignore_timestamps`, or a `diarize` value other than those listed, is `invalid_config`. +A value of the wrong type for `ignore_timestamps`, or a `diarize` value other than those listed, is `invalid_config`. So is an unknown field, including a misspelled one; the error's `message` names the field. `invalid_config` closes the connection with 4400. Send only the fields listed here: not every other field is refused, so check `session_started.config`, which shows the settings in effect. #### Audio format @@ -162,6 +164,7 @@ The default, the allowed range (inclusive) and the meaning depend on the mode: | `vad` | 700 | 300–60000 | The silence that ends a turn (`reason: "eot"`). | | `manual` | 60000 | 300–60000 | A safety net: a turn silent for this long ends with `reason: "silence_timeout"`. At 60000 it never fires. | +- In `vad` mode, every pause of this length ends the turn, including a pause inside a sentence. At short values, one sentence can arrive as several turns, and a word that starts softly right after such a pause can be split across two turns; both happen more often the shorter the value. For conversational speech, start with 500 ms or more. - If you leave it out, the mode's default applies. `session_started` and `configured` always show the value in effect. - The value must be a whole number inside the range of the mode it will run under: the `mode` in the same message, or else the current mode. In `config` the current mode is `auto`, so a `config` with `silence_timeout_ms: 700` and no `mode` is refused. - Outside the range, `config` gets `error` with `invalid_config` and the connection closes with 4400; the message gives the range, for example "turn_detection.silence_timeout_ms must be between 3000 and 60000 in auto mode". A `configure` gets `configured` with `ok: false` and the same message, and nothing in that `configure` is applied. @@ -175,7 +178,9 @@ The default, the allowed range (inclusive) and the meaning depend on the mode: { "type": "audio", "data": "" } ``` -Audio in the format from `config`, as MessagePack `bin`. Send audio only after `session_started`. For live audio, send it as it is captured; 80–100 ms per message is a good size. You can send faster than real time, for example from a file: the server accepts it and reads it at a bounded rate, and results keep following the audio. +Audio in the format from `config`. `data` must be MessagePack `bin` holding the raw PCM bytes. Do not base64-encode the audio, and do not send it as a string or an array of numbers. If you do, you receive a recoverable `invalid_request`, and those messages are dropped: nothing in them is transcribed or counted as received audio. + +Send audio after `session_started` (see [`config`](#config) for audio sent earlier). For live audio, send it as it is captured; 80–100 ms per message is a good size. You can send faster than real time, for example from a file: the server accepts it and reads it at a bounded rate, and results keep following the audio. ### `finalize` @@ -198,7 +203,9 @@ Commit the current turn now, with `reason: "finalize"`. In `auto` mode, the serv } ``` -Change settings during the session. Every field is optional and takes the same values as in `config`; a field you leave out keeps its current value. `context` accepts `terms` only, and the new list replaces the old one; `context.history` is ignored in `configure`, so send earlier lines with [`context_add`](#context_add). The server answers with [`configured`](#configured). If any value is invalid, nothing in the message is applied. +Change settings during the session. `turn_detection`, `results` and `context` take the same values as in `config`; every field is optional, and a field you leave out keeps its current value. `context` accepts `terms` only, and the new list replaces the old one; send earlier lines with [`context_add`](#context_add). The server answers with [`configured`](#configured). If any value is invalid, nothing in the message is applied. + +`audio` is fixed for the session: to change the audio format, open a new session. A `configure` with an unknown field, with an `audio` that differs from the session's, or with `context.history` gets `configured` with `ok: false`, and nothing in it is applied. An `audio` identical to the session's is accepted. As with `config`, send only the fields listed here; `configured.config` shows the settings in effect. - A change to `ignore_timestamps` or `diarize` applies to turns that open after `configured`. A turn that is already open keeps the settings it started with, so its `eager_end_of_turn` and `committed` always agree. - When `mode` changes to or from `auto`, the server first settles its turn decisions on the audio already sent, so `configured` can take a few seconds. @@ -223,7 +230,7 @@ Add conversation context. The server keeps the session's conversation and uses t { "type": "keepalive" } ``` -Keeps an idle session open. A session closes after 30 seconds without any client message. +Keeps an idle session open. A session closes after 30 seconds without any client message. A WebSocket ping counts as client activity too, so a client library that sends pings on its own keeps the session open. ### `audio_done` @@ -369,7 +376,7 @@ With `results.eot_events: true`, for display only. In `vad` mode it is 0.7 × th | `config` | map | The configuration in effect after the `configure`. | | `message` | string | Present when `ok` is `false`: why, in English, for humans. | -The reply to `configure`. `ok: false` comes for an invalid value, such as a `silence_timeout_ms` outside the mode's range, or a value over a limit. +The reply to `configure`. `ok: false` comes for an invalid value, such as a `silence_timeout_ms` outside the mode's range, or a value over a limit, and for an unknown field, a changed `audio` or `context.history`. ### `progress` @@ -423,15 +430,23 @@ In every mode: ## Speakers -`text` can contain speaker tags `<|speaker:N|>` (N = 0, 1, ...). Numbers are stable within a session and start over in a new one. Strip the tags if you only need the words, or send `results.diarize: false` to get text without them. With `diarize: false`, whitespace in the text, line breaks included, is collapsed to single spaces and trimmed; sound tags in square brackets, such as `[laughter]`, stay. +`text` can contain speaker tags `<|speaker:N|>` (N = 0, 1, ...). Numbers are stable within a session and start over in a new one. Strip the tags if you only need the words, or send `results.diarize: false` to get text without them. With `diarize: false`, whitespace in the text, line breaks included, is collapsed to single spaces and trimmed; sound tags stay (see [Sound tags](#sound-tags)). + +Whitespace right after a speaker tag is not significant and can vary, for example `<|speaker:0|>Can you` or `<|speaker:0|> Can you`. To get each speaker's text, split the text on the tags, then trim each segment. If you only need the words, remove the tags, then trim the text and collapse runs of whitespace into one space. + +## Sound tags + +`text` can contain sound tags in square brackets, such as `[laughter]`, `[laughs]`, `[sighs]` or `[pause]`, in every turn detection mode and whether `diarize` is on or off. The set is open: a tag can be any short description, sometimes several words long. A turn's text can be a tag alone. Language detection ignores tags. + +Tags are sounds, not words. Strip them when you need only the words, for example for search, word counts, or deciding whether there is anything to answer: remove every match of the regular expression `\[[^\[\]]*\]`, then trim the text and collapse runs of whitespace into one space. If you keep them for a language model, tell it that they are sounds, and do not answer a turn that holds only tags. ## Language -`draft`, `eager_end_of_turn` and `committed` carry the language of their own `text`. Use `language_code` (ISO 639-1) in your code; `language` is its English name, for display, with the same names as [`POST /v1/asr`](/api-reference/endpoint/openapi-v1/speech-to-text). The language is detected from that message's text alone, ignoring speaker tags, sound tags and interjections such as "okay" or "hmm". It is one of: +`draft`, `eager_end_of_turn` and `committed` carry the language of their own `text`. Use `language_code` (ISO 639-1) in your code; `language` is its English name, for display, with the same names as [`POST /v1/asr`](/api-reference/endpoint/openapi-v1/speech-to-text). The language is detected from that message's text alone, ignoring speaker tags, sound tags and interjections such as "okay", "hmm" or "uh". It is one of: `ar`, `zh`, `cs`, `da`, `nl`, `en`, `fi`, `fr`, `de`, `el`, `hi`, `hu`, `id`, `it`, `ja`, `ko`, `mk`, `ms`, `fa`, `pl`, `pt`, `ro`, `ru`, `es`, `sv`, `tl`, `th`, `tr`, `vi`. -Both fields are omitted when the text is empty, too short to tell (most replies under three words, such as "Okay" or "Yes.") or ambiguous; keep the last label you saw. Mixed-language text gets one label, the dominant language. Drafts are labeled one by one and may change; `committed` is final. +Both fields are omitted when the text is empty, too short to tell or ambiguous. Replies such as "Okay" or "Yes." usually get no label, and other text of only a few words often does not get one either; keep the last label you saw. Mixed-language text gets one label, the dominant language. Drafts are labeled one by one and may change; `committed` is final. ## Degraded commits @@ -439,19 +454,19 @@ If the final pass for a turn fails, you receive `error` with `engine_error` and ## Errors and close codes -| `error_type` | `code` | Recoverable | Then | -| ----------------------- | --------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `invalid_config` | 400 | No | Close 4400. A bad or missing `config`, a value over a limit, or `silence_timeout_ms` outside the mode's range. | -| `unsupported_audio` | 400 | No | Close 4400. A sample rate not in the list, a format other than `pcm_s16le`, or more than one channel. | -| `invalid_request` | 400 | Yes | The session continues. A text frame, an unreadable message, an over-limit `context_add`, a second `config`, or `agent_speaking` without a boolean `speaking`. | -| `idle_timeout` | 408 | No | Close 4408 after 30 seconds without any client message. | -| `max_duration_exceeded` | 413 | No | The open turn is committed (`reason: "finalize"`), then close 4413. | -| `frame_too_large` | 413 | No | Close 1009. A message over `limits.max_frame_bytes`. | -| `engine_error` | 500, 503 or 504 | Yes | A [degraded `committed`](#degraded-commits) follows (503: temporarily unavailable; 504: timed out). If the session could not start: `recoverable: false`, close 4500. | -| `internal` | 500 | No | Close 4500. | -| `server_restarting` | 503 | Yes | The open turn is committed (`reason: "finalize"`), then close 1012. Reconnect for a new session. | +| `error_type` | `code` | Recoverable | Then | +| ----------------------- | --------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `invalid_config` | 400 | No | Close 4400. The first message is not a readable `config` (for example a text frame or JSON), or the `config` is invalid, for example with an unknown field, a value over a limit, or `silence_timeout_ms` outside the mode's range. | +| `unsupported_audio` | 400 | No | Close 4400. A sample rate not in the list, a format other than `pcm_s16le`, or more than one channel. | +| `invalid_request` | 400 | Yes | The session continues. After `config`: a text frame, an unreadable message, an `audio` message whose `data` is not `bin` (the message is dropped), an over-limit `context_add`, a second `config`, or `agent_speaking` without a boolean `speaking`. | +| `idle_timeout` | 408 | No | Close 4408: 30 seconds without any client message, or no `config` within 10 seconds of connecting. | +| `max_duration_exceeded` | 413 | No | The open turn is committed (`reason: "finalize"`), then close 4413. | +| `frame_too_large` | 413 | No | Close 1009. A message over `limits.max_frame_bytes`. | +| `engine_error` | 500, 503 or 504 | Yes | A [degraded `committed`](#degraded-commits) follows (503: temporarily unavailable; 504: timed out). If the session could not start: `recoverable: false`, close 4500. | +| `internal` | 500 | No | Close 4500. | +| `server_restarting` | 503 | Yes | The open turn is committed (`reason: "finalize"`), then close 1012. Reconnect for a new session. | -`server_restarting` is the one recoverable error that closes the connection: the session cannot go on, but you can reconnect at once for a new one. If the first message is not a readable `config`, you get an `error` with `recoverable: false` and close 4400. +`server_restarting` is the one recoverable error that closes the connection: the session cannot go on, but you can reconnect at once for a new one. If the first message is not a readable `config`, you get `invalid_config` with `recoverable: false` and close 4400. A normal end is `session_done`, then close 1000. Match on `error_type`, not on `message`. In `auto` mode, a turn the server commits before closing may have `user_turn: false`. @@ -464,7 +479,7 @@ A normal end is `session_done`, then close 1000. Match on `error_type`, not on ` | Turn length | 60 seconds (`limits.max_turn_ms`); 20 seconds for an `auto` turn without `user_turn_taken` | | `silence_timeout_ms` | `auto`: 3000–60000; `vad` and `manual`: 300–60000 | | Message size | 1 MiB (`limits.max_frame_bytes`) | -| First `config` | Within 10 seconds of connecting | +| First `config` | Within 10 seconds of connecting; otherwise `idle_timeout` and close 4408 | | Idle | 30 seconds without any client message | | Drafts | At most every 250 ms | | Context | 32 items × 2,000 characters per message; 100 terms × 32 characters | @@ -509,7 +524,7 @@ Turn 0 is the user's turn. The agent prepared its reply on `eager_end_of_turn`; ## Compatibility rules -- **Ignore fields and message types you do not know.** Version 1 only ever adds them. Breaking changes get a new path (`/v2/...`). +- **Ignore fields and message types you do not know in server messages.** Version 1 only ever adds them. Breaking changes get a new path (`/v2/...`). - Treat an unknown `reason` like `eot`, and an unknown `turn_kind` like `"other"` when `user_turn` is `false`. - Match errors on `error_type`, not on `message`. diff --git a/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx b/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx index f3a40fa..cf5f3e5 100644 --- a/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx +++ b/developer-guide/sdk-guide/cookbook/realtime-transcription.mdx @@ -241,6 +241,10 @@ ws.on("close", (code, reason) => console.log(`closed: ${code} ${reason}`)); +The audio goes in `data` as the raw PCM bytes, which must travel as MessagePack `bin`, not as base64 text: msgpack for Python packs `bytes` as `bin` by default (do not set `use_bin_type=False`), and `@msgpack/msgpack` packs a `Buffer` or `Uint8Array` as `bin`. + +The server closes a session after 30 seconds without any client message, but a WebSocket ping counts as one, and Python's `websockets` sends a ping every 20 seconds by default, so the session stays open even if your audio stops; see [Capture audio well](/developer-guide/sdk-guide/cookbook/realtime-voice-agent#capture-audio-well) for a live source that can stall. + Run it with your API key in `FISH_API_KEY`: ```bash @@ -254,25 +258,25 @@ For a 15-second customer-support clip, the output looks like this (`...` marks d ```text session s_ab12cd34 (request 0b6f4c1e-7d2a-4e8b-9f3c-5a1d2e3f4b6c) speech_start #0 - draft #0: <|speaker:0|>Can you... - draft #0: <|speaker:0|>Can you tell + draft #0: <|speaker:0|> Can you... + draft #0: <|speaker:0|> Can you tell user_turn_taken #0 - draft #0: <|speaker:0|>Can you tell me what... + draft #0: <|speaker:0|> Can you tell me what... ... - draft #0: <|speaker:0|>Can you tell me what time the pharmacy on Main Street closes tonight? - eager #0: <|speaker:0|>Can you tell me what time the pharmacy on Main Street closes tonight? -committed #0 [user ] 2768-8000 ms (eot, en): <|speaker:0|>Can you tell me what time the pharmacy on Main Street closes tonight? + draft #0: <|speaker:0|> Can you tell me what time the pharmacy on Main Street closes tonight? + eager #0: <|speaker:0|> Can you tell me what time the pharmacy on Main Street closes tonight? +committed #0 [user ] 2768-8000 ms (eot, en): <|speaker:0|> Can you tell me what time the pharmacy on Main Street closes tonight? speech_start #1 - draft #1: <|speaker:0|>Uh-huh. + draft #1: <|speaker:0|> Uh-huh. user_turn_taken #1 - eager #1: <|speaker:0|>Uh-huh. -committed #1 [user ] 9840-11840 ms (eot, ?): <|speaker:0|>Uh-huh. + eager #1: <|speaker:0|> Uh-huh. +committed #1 [user ] 9840-11840 ms (eot, ?): <|speaker:0|> Uh-huh. speech_start #2 - draft #2: <|speaker:0|>I'd like + draft #2: <|speaker:0|> I'd like user_turn_taken #2 ... - draft #2: <|speaker:0|>I'd like to change my deli- -committed #2 [user ] 13168-14992 ms (audio_done, en): <|speaker:0|>I'd like to change my delivery + draft #2: <|speaker:0|> I'd like to change my deli- +committed #2 [user ] 13168-14992 ms (audio_done, en): <|speaker:0|> I'd like to change my delivery done: 15000 ms of audio, 3 turns closed: 1000 done ``` @@ -285,7 +289,7 @@ The clip ends mid-sentence, so the last turn is committed by `audio_done`. Draft - **`draft`** is the whole turn so far, and replaces the previous draft. Drafts come only with `results.drafts: true`; use them for live captions. - **`user_turn_taken`** says the server judged that the user is taking their turn, rather than noise or someone else nearby. - **`eager_end_of_turn`** comes with `turn_detection.eager: true` when the turn has probably ended. Its text equals the final transcript unless `turn_resumed` follows, so an agent can start its reply early. -- **`committed`** is the final transcript of the turn. `user_turn: false` marks a non-user turn, such as background noise or a short acknowledgement while your agent is speaking. Speaker tags such as `<|speaker:0|>` are stable within the session; strip them, or send `results.diarize: false`. `language_code` is omitted when the text is too short to tell, as for "Uh-huh." above. +- **`committed`** is the final transcript of the turn. `user_turn: false` marks a non-user turn, such as background noise or a short acknowledgement while your agent is speaking. Speaker tags such as `<|speaker:0|>` are stable within the session; strip them and trim the text, or send `results.diarize: false`. Text can also contain [sound tags](/api-reference/endpoint/websocket/asr-realtime#sound-tags) such as `[laughter]`. `language_code` is omitted when the text is too short to tell, as for "Uh-huh." above. - **`session_done`** follows `audio_done`, and the server then closes the connection with code 1000. Ignore message types and fields you do not know: new ones can be added. Every message is described in the [WebSocket reference](/api-reference/endpoint/websocket/asr-realtime#server-messages). @@ -401,7 +405,12 @@ registerProcessor("pcm16", Pcm16);`; const RATES = [8000, 12000, 16000, 22050, 24000, 32000, 44100, 48000]; const $ = id => document.getElementById(id); - const strip = text => (text || "").replace(/<\|[^|]*\|>/g, "").trim(); // speaker tags + // Speaker tags and sound tags out, whitespace collapsed. + const strip = text => + (text || "") + .replace(/<\|[^|]*\|>|\[[^\[\]]*\]/g, "") + .replace(/\s+/g, " ") + .trim(); let ws, ctx, stream, diff --git a/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx b/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx index fe9ceca..1b040f8 100644 --- a/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx +++ b/developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx @@ -77,7 +77,7 @@ import websockets URL = "wss://api.fish.audio/v1/asr/realtime" CHUNK_MS = 100 -TAGS = re.compile(r"<\|[^|]*\|>") # speaker tags such as <|speaker:0|> +TAGS = re.compile(r"<\|[^|]*\|>|\[[^\[\]]*\]") # speaker tags such as <|speaker:0|>, sound tags such as [laughter] T0 = time.monotonic() @@ -86,8 +86,8 @@ def log(*args): def words(text): - """The transcript without speaker tags.""" - return TAGS.sub("", text or "").strip() + """The transcript without speaker or sound tags, whitespace collapsed.""" + return " ".join(TAGS.sub("", text or "").split()) async def think(history): @@ -238,7 +238,7 @@ The recipe keeps a reply's full text in its history even when the user interrupt ## Decide when to answer -Answer a turn when `committed.user_turn` is `true` and its text, without speaker tags, is not empty. Do not decide by `reason`: a turn can end for several reasons, and non-user turns end with `eot` too. If `user_turn` is ever missing, treat it as `true`. +Answer a turn when `committed.user_turn` is `true` and its text, without speaker tags and [sound tags](/api-reference/endpoint/websocket/asr-realtime#sound-tags), is not empty. Do not decide by `reason`: a turn can end for several reasons, and non-user turns end with `eot` too. If `user_turn` is ever missing, treat it as `true`. A turn with `user_turn: false` is not the user talking to your agent: usually background noise, a TV or another person nearby (`turn_kind: "other"`), or an acknowledgement while your agent was speaking (`turn_kind: "backchannel"`). Its text can be shown dimmed in a transcript, or left out. @@ -273,7 +273,7 @@ While your agent is speaking: 2. On `user_backchannel`, the user only acknowledged ("mm-hm", "yeah", "right", "uh-huh"): restore the volume and keep talking. 3. On `user_turn_taken`, the user is taking the turn: stop the agent and send `agent_speaking: false`. 4. On `committed` with `user_turn: false`, restore the volume. -5. Cap any duck on your side to a few seconds: a turn over continuous background noise can last up to 20 seconds. When the user starts talking over such noise, `user_turn_taken` can come a few hundred milliseconds later than usual. +5. Cap any duck on your side at about 2 seconds, then restore the volume even if the turn is still open. For a real interruption, `user_turn_taken` normally comes within about a second of `speech_start`. Non-user turns can stay open much longer: leftover echo of your agent's own voice can open a turn that lasts as long as the reply, and a turn over continuous background noise can last up to 20 seconds. When the user starts talking over such noise, `user_turn_taken` can come a few hundred milliseconds later than usual. In `vad` and `manual` mode there is no `user_turn_taken` or `user_backchannel`: stop on `speech_start` instead. @@ -306,15 +306,16 @@ The server uses the conversation to recognize names and follow-ups: ## Capture audio well -- **Keep echo cancellation on.** Without it, the agent's own voice reaches the microphone, opens turns and can be taken for the user. In a browser, request `echoCancellation: true` from `getUserMedia`. +- **Keep echo cancellation on.** Without it, the agent's own voice reaches the microphone, opens turns and can be taken for the user. In a browser, request `echoCancellation: true` from `getUserMedia`. Some echo can remain even with echo cancellation. It is usually committed as a non-user turn, but it still opens turns (`speech_start`) while your agent speaks, which is why ducks need a cap. If the start of most replies is ducked, echo is leaking: check echo cancellation, or test with headphones. - **Turn automatic gain control off where you can** (`autoGainControl: false` in a browser), so quiet background sound is not amplified between phrases. - **Test with headphones first.** They keep the agent's voice out of the microphone, so you can check turn-taking before you tune echo cancellation. - **Send audio as you capture it**, mono 16-bit PCM in 80–100 ms messages, at the device's own rate if it is one of the supported rates. The server converts the rate, so do not resample. Telephony audio at 8 kHz is supported; decode μ-law or A-law call audio to 16-bit PCM first. +- **Watch your audio source.** The server closes a session after 30 seconds without any client message (`idle_timeout`, close 4408), but a WebSocket ping counts as one, and Python's `websockets` sends a ping every 20 seconds by default. With that default, a session stays open even when your audio stops, and it counts against your concurrency limit until you close it or the 60-minute session limit ends it. If your audio source can stall, watch it in your own code and send `audio_done` or close the connection when it stops, or pass `ping_interval=None` to `websockets.connect` to leave idle detection to the server. Node's `ws` sends no pings by default. - **Keep keys on your server.** In a browser, capture audio and relay it through your server, as in [Capture from a browser microphone](/developer-guide/sdk-guide/cookbook/realtime-transcription#capture-from-a-browser-microphone). Send `agent_speaking` on that same connection so it stays in order with the audio. ## Languages -Turn detection, including eager end of turn, backchannel detection and background awareness, is trained on a set of major languages. Transcription and language identification cover many more languages. In languages outside that set, a turn ends about 2 seconds after the speaker stops and there is no `eager_end_of_turn`. For faster turns in those languages, use `vad` mode with a shorter `silence_timeout_ms`, or end turns yourself with `finalize`. +Turn detection, including eager end of turn, backchannel detection and background awareness, works on speech in every language. In some languages, a turn can end a little later after the speaker stops. For faster turns, use `vad` mode (500 ms or more is a good start for `silence_timeout_ms`), or end turns yourself with `finalize`. ## Connect your LLM and speech diff --git a/features/speech-to-text/realtime.mdx b/features/speech-to-text/realtime.mdx index 3928fd3..9188827 100644 --- a/features/speech-to-text/realtime.mdx +++ b/features/speech-to-text/realtime.mdx @@ -119,15 +119,26 @@ The SDKs do not wrap this endpoint; you connect with any WebSocket client and ex ## How a session works -Audio is divided into turns. A turn opens when voice activity begins (`speech_start`) and ends with exactly one `committed` message, which holds the final transcript and says why the turn ended. In between, you can receive live `draft` text and turn signals: +Audio is divided into turns. A turn opens when voice activity begins (`speech_start`), usually before the server knows what the sound is: the user speaking, a short acknowledgement, noise or another voice. It ends with exactly one `committed` message, which holds the final transcript and says why the turn ended. In between, you can receive live `draft` text and turn signals that tell you what the turn is. In the default `auto` mode, with `results.drafts` and `turn_detection.eager` turned on as in the example above: ```text -speech_start → draft … → user_turn_taken → draft … → eager_end_of_turn → committed +The user's turn: + speech_start → draft … → user_turn_taken → draft … → eager_end_of_turn → committed (user_turn: true) + +"Mm-hm" from the user while your agent speaks: + speech_start → draft … → user_backchannel → draft … → committed (user_turn: false, turn_kind: "backchannel") + +Noise, a TV or another voice: + speech_start → draft … → committed (user_turn: false, turn_kind: "other") ``` +- `draft`: the turn's text so far. Drafts come in every kind of turn, so a draft does not mean the user is talking. - `user_turn_taken`: the server judged that the user is taking their turn. -- `eager_end_of_turn`: the turn has probably ended; its text equals the final transcript unless `turn_resumed` follows. -- `committed`: the final transcript. `user_turn` says whether it was the user's turn. +- `user_backchannel`: while your agent is speaking (see [Backchannel detection](#backchannel-detection)), the user only acknowledged; keep talking. If the user goes on to take the turn, `user_turn_taken` follows, and the turn that holds the acknowledgement is the user's. That can be a new turn, with its own `speech_start`. +- `eager_end_of_turn`: the turn has probably ended; its text equals the final transcript unless `turn_resumed` follows. Only the user's turns get it, and not every one does. +- `committed`: the final transcript. `user_turn` says whether it was the user's turn, and `turn_kind` what it was otherwise. + +So do not stop your agent on `speech_start` or on a `draft`: noise opens turns and gets drafts too. Wait for `user_turn_taken`. In `vad` and `manual` mode there is no `user_turn_taken`: every turn is a user turn, so stop on `speech_start` there (see [Turn detection modes](#turn-detection-modes)). ## Smart Turn Detection @@ -170,9 +181,9 @@ Eager end of turn, backchannel detection and background awareness are available ## Languages -Smart Turn Detection is trained on a set of major languages. Transcription covers many more languages, and `draft`, `eager_end_of_turn` and `committed` carry the language of their own text as `language_code` when it can be identified, one of 29 [supported codes](/api-reference/endpoint/websocket/asr-realtime#language). +Smart Turn Detection, including eager end of turn, backchannel detection and background awareness, works on speech in every language. In some languages, a turn can end a little later after the speaker stops. -In languages outside that set, a turn ends about 2 seconds after the speaker stops, and there is no eager signal. For faster turns in those languages, use `vad` mode, or end turns yourself with `finalize`. +`draft`, `eager_end_of_turn` and `committed` carry the language of their own text as `language_code` when it can be identified, one of 29 [supported codes](/api-reference/endpoint/websocket/asr-realtime#language). ## Latency @@ -192,7 +203,7 @@ Send 16-bit little-endian PCM, mono, at 8000, 12000, 16000, 22050, 24000, 32000, ### Speakers -Transcripts contain speaker tags such as `<|speaker:0|>`, stable within a session. Strip them, or send `results.diarize: false` for plain text. +Transcripts contain speaker tags such as `<|speaker:0|>`, stable within a session. Strip them, or send `results.diarize: false` for plain text. Whitespace right after a tag can vary, so trim the text after you strip the tags. Transcripts can also contain sound tags such as `[laughter]` or `[pause]`; see [Sound tags](/api-reference/endpoint/websocket/asr-realtime#sound-tags). ### Keys and browsers @@ -200,7 +211,7 @@ Your API key travels in a request header. Connect from your server, never from a ### Limits -A session lasts up to 60 minutes and a turn up to 60 seconds. Each open connection holds one of your team's concurrent session slots. See [Limits](/api-reference/endpoint/websocket/asr-realtime#limits) for the full list. +A session lasts up to 60 minutes and a turn up to 60 seconds. Each open connection counts against the same concurrency limit as your other API requests, from the moment it opens until it closes, even while you send no audio; see [Concurrent request limits](/developer-guide/models-pricing/pricing-and-rate-limits#concurrent-request-limits). See [Limits](/api-reference/endpoint/websocket/asr-realtime#limits) for the full list. ## Going further