diff --git a/developer-guide/getting-started/changelog.mdx b/developer-guide/getting-started/changelog.mdx index f01b7d86..f27cf362 100644 --- a/developer-guide/getting-started/changelog.mdx +++ b/developer-guide/getting-started/changelog.mdx @@ -21,7 +21,7 @@ rss: true Use model ID `drama-3-preview` in the API. Behavior and availability may change while it is in preview. - [Text to Speech API](/api-reference/endpoint/openapi-v1/text-to-speech) + [Drama 3 guide](/features/text-to-speech/drama-3) | [Text to Speech API](/api-reference/endpoint/openapi-v1/text-to-speech) diff --git a/docs.json b/docs.json index fe2b8336..3005692e 100644 --- a/docs.json +++ b/docs.json @@ -33,6 +33,7 @@ "icon": "microphone", "pages": [ "features/text-to-speech", + "features/text-to-speech/drama-3", "developer-guide/core-features/emotions", { "group": "Fine-grained Control", diff --git a/features/text-to-speech.mdx b/features/text-to-speech.mdx index 3d109f42..a4a502b8 100644 --- a/features/text-to-speech.mdx +++ b/features/text-to-speech.mdx @@ -110,6 +110,7 @@ curl --request POST https://api.fish.audio/v1/tts \ - **`s2.1-pro`**: recommended for production, with improved quality, latency, and throughput over S2-Pro. - **`s2.1-pro-free`**: the same model at $0 for testing, prototyping, development, and smaller businesses, without TTFA or DPA guarantees. +- **`drama-3-preview`**: preview model you direct in plain language, with sounds, pauses, and paired tags for specific words. See [Drama 3](/features/text-to-speech/drama-3). - **`s2-pro`**: previous-generation S2 model with multi-speaker and natural-language expression control. - **`s1`**: previous generation, `(parenthesis)` emotion tags. S1 is deprecated and will be retired on December 31, 2026. After that date, requests that specify `s1` are served by `s2.1-pro`. See [Migrate from S1](/developer-guide/models-pricing/migrate-from-s1). diff --git a/features/text-to-speech/drama-3.mdx b/features/text-to-speech/drama-3.mdx new file mode 100644 index 00000000..e25b209c --- /dev/null +++ b/features/text-to-speech/drama-3.mdx @@ -0,0 +1,203 @@ +--- +title: "Drama 3" +description: "Text-to-speech model you can direct" +icon: "clapperboard" +tag: "Preview" +--- + +import Support from "/snippets/support.mdx"; +import { AudioClip } from "/snippets/audio-clip.jsx"; + + + Drama 3 is in preview. Behavior can still change. + + +Drama 3 is our latest text-to-speech model built for voice direction and content creation. Alongside your script, you describe how each line should be performed, add sounds and pauses, and limit a delivery to specific words. + +## What's new + +- Direct a performance in plain language, including a language other than the script. +- Add a sound or a pause where it happens, or limit a delivery to specific words. +- Switch language for a phrase, or pin a pronunciation with phoneme tags. +- Keep your current `POST /v1/tts` request and set the model header to `drama-3-preview`. + +## Voice cast + +Choosing a voice that fits the role is key to getting the best out of Drama 3. These public voices are used in the examples on this page: + +| Name | Language | ID | +|---|---|---| +| [Adrian](https://fish.audio/app/text-to-speech/?modelId=bf322df2096a46f18c579d0baa36f41d) | EN-US, male | `bf322df2096a46f18c579d0baa36f41d` | +| [Delia](https://fish.audio/app/text-to-speech/?modelId=c70cfb98867a4bfb9ae20d2401257b92) | EN-US, female | `c70cfb98867a4bfb9ae20d2401257b92` | +| [Sheila](https://fish.audio/app/text-to-speech/?modelId=f3a2b90078d54a65af4b95f63bc7798e) | EN-US, female | `f3a2b90078d54a65af4b95f63bc7798e` | + +## How to use + +To use Drama 3, set the `model` header to `drama-3-preview` on your existing `POST /v1/tts` request. + +| Status | Model header | Availability | +|---|---|---| +| Preview | `drama-3-preview` | Available now | + + + If the `model` header is missing or misspelled, the request doesn't fail: it + falls back to `s2.1-pro`. + + + +## Direct the performance + +Directions are plain text in your script, and you can mix them freely in the same line. They work the same way in API requests and in the web app. + +### Describe the delivery in your own words + +Put a direction in square brackets before the words it applies to. Write it the way you'd brief a voice actor: emotion, intent, pacing, or who the character is talking to. + + + +### Add sounds and pauses + +Put a sound or pause in brackets exactly where it should happen, such as a sigh, a laugh, a breath, or a short pause. + + + +### Write directions in any language + +Directions can be in English or your own language, and they don't have to match the language of the script. + + + +### Change only specific words + +Wrap words in a paired tag, such as `…`, to change the delivery of those words only. Supported tags: ``, ``, ``, ``, ``, ``, ``. + + + +### Switch language mid-line (Experimental) + +Wrap a phrase in a language tag to switch language for those words only. This is useful for a tutor, a conversation, or any line that mixes two languages. + +Examples: + + + + + + + + + + + + + + + +- Use a voice cloned from a speaker fluent in both languages. +- Wrap only the words that change language. +- Try the pair you need. Accuracy varies by language and voice. + + + Language tags are experimental. They are not guaranteed for every language or voice, and the model may switch language on its own when the text is already multilingual. + + +## Pronunciation + +Drama 3 works with the phoneme tags already supported on other models. To pin down how a name or term is pronounced, wrap its phonemes in `<|phoneme_start|>` and `<|phoneme_end|>`. In English (CMU Arpabet), the digit on a vowel is its stress: `1` is primary and `0` is unstressed. These two takes are the same word and the same voice, with the stress on a different syllable. + + + + + +The symbol set depends on the language: + +| Language | Phonemes | +|---|---| +| [English](/developer-guide/core-features/fine-grained-control/english) | CMU Arpabet | +| [Chinese](/developer-guide/core-features/fine-grained-control/chinese) | pinyin | +| [Japanese](/developer-guide/core-features/fine-grained-control/japanese) | romaji | + +## Multi-speaker dialogue + +Drama 3 works with multi-speaker dialogue the same way current models do. Generate a whole scene in one request: pass one voice ID per speaker as an array in `reference_id`, and start each turn with a speaker tag. `<|speaker:0|>` uses the first voice, `<|speaker:1|>` the second, and so on. Directions and paired tags from this page work inside each turn. See `reference_id` in the [Text to Speech API](/api-reference/endpoint/openapi-v1/text-to-speech). + +This scene uses three public voices: [Delia](https://fish.audio/app/text-to-speech/?modelId=c70cfb98867a4bfb9ae20d2401257b92) narrates, [Adrian](https://fish.audio/app/text-to-speech/?modelId=bf322df2096a46f18c579d0baa36f41d) is the captain, and [Sheila](https://fish.audio/app/text-to-speech/?modelId=f3a2b90078d54a65af4b95f63bc7798e) is the engineer. + +```text +Narrator: [calm, measured storyteller voice] The ship had been silent for three days. +Captain: [low, gravelly, tired but in control] Status report. +Engineer: [fast, nervous, stumbling over words] Uh, sir, so, the thing is, the reactor is, it is not, it is not great. +Captain: [slow, dangerously quiet] Define "not great." +Engineer: [voice breaking, close to panic] It is going to blow. +``` + +The names in the script above are for reading. The request sends the same lines as one `text` string, with a speaker tag instead of a name: `<|speaker:0|>` is the narrator, `<|speaker:1|>` is the captain, and `<|speaker:2|>` is the engineer. + +```bash +curl --fail --show-error --request POST https://api.fish.audio/v1/tts \ + --header "Authorization: Bearer $FISH_API_KEY" \ + --header "Content-Type: application/json" \ + --header "model: drama-3-preview" \ + --data '{ + "text": "<|speaker:0|>[calm, measured storyteller voice] The ship had been silent for three days.<|speaker:1|>[low, gravelly, tired but in control] Status report.<|speaker:2|>[fast, nervous, stumbling over words] Uh, sir, so, the thing is, the reactor is, it is not, it is not great.<|speaker:1|>[slow, dangerously quiet] Define \"not great.\"<|speaker:2|>[voice breaking, close to panic] It is going to blow.", + "reference_id": ["c70cfb98867a4bfb9ae20d2401257b92", "bf322df2096a46f18c579d0baa36f41d", "f3a2b90078d54a65af4b95f63bc7798e"], + "format": "mp3" + }' \ + --output dialogue.mp3 +``` + + + + diff --git a/llms.txt b/llms.txt index 89f8ac8c..43a83d04 100644 --- a/llms.txt +++ b/llms.txt @@ -42,6 +42,7 @@ ## Product Guides - [Text to Speech Guide](https://docs.fish.audio/developer-guide/core-features/text-to-speech.md): Convert text to natural-sounding speech with Fish Audio. +- [Drama 3](https://docs.fish.audio/features/text-to-speech/drama-3.md): Preview text-to-speech model (drama-3-preview) directed with plain-language directions in square brackets, sounds and pauses, paired tags for specific words, and multi-speaker dialogue. - [Speech to Text Guide](https://docs.fish.audio/features/speech-to-text.md): Transcribe audio with transcribe-1-pro, including multi-speaker conversations with speaker turns, long recordings and inline emotion cues. - [Voice Design Guide](https://docs.fish.audio/features/voice-design.md): Generate candidate voices from natural-language prompts. - [Creating Voice Models](https://docs.fish.audio/developer-guide/core-features/creating-models.md): Learn how to create custom voice models with Fish Audio. diff --git a/snippets/audio-clip.jsx b/snippets/audio-clip.jsx new file mode 100644 index 00000000..f0efbe37 --- /dev/null +++ b/snippets/audio-clip.jsx @@ -0,0 +1,113 @@ +import { useEffect, useRef, useState } from 'react'; + +export const AudioClip = ({ src, title, text }) => { + const [isPlaying, setIsPlaying] = useState(false); + const [currentTime, setCurrentTime] = useState(0); + const [duration, setDuration] = useState(0); + const audioRef = useRef(null); + + useEffect(() => { + const audio = audioRef.current; + if (!audio) return; + + const updateTime = () => setCurrentTime(audio.currentTime); + const updateDuration = () => setDuration(audio.duration); + const handlePlay = () => setIsPlaying(true); + const handlePause = () => setIsPlaying(false); + + audio.addEventListener('timeupdate', updateTime); + audio.addEventListener('loadedmetadata', updateDuration); + audio.addEventListener('play', handlePlay); + audio.addEventListener('pause', handlePause); + audio.addEventListener('ended', handlePause); + + return () => { + audio.removeEventListener('timeupdate', updateTime); + audio.removeEventListener('loadedmetadata', updateDuration); + audio.removeEventListener('play', handlePlay); + audio.removeEventListener('pause', handlePause); + audio.removeEventListener('ended', handlePause); + }; + }, []); + + const togglePlay = () => { + const audio = audioRef.current; + if (!audio) return; + if (audio.paused) { + document.querySelectorAll('audio').forEach((other) => { + if (other !== audio) other.pause(); + }); + audio.play().catch(() => setIsPlaying(false)); + } else { + audio.pause(); + } + }; + + const handleProgressChange = (event) => { + const next = parseFloat(event.target.value); + audioRef.current.currentTime = next; + setCurrentTime(next); + }; + + const formatTime = (time) => { + if (!Number.isFinite(time)) return '0:00'; + const minutes = Math.floor(time / 60); + const seconds = Math.floor(time % 60); + return `${minutes}:${seconds.toString().padStart(2, '0')}`; + }; + + return ( +
+ {title || text ? ( +
+ {title ?

{title}

: null} + {text ? ( +

+ {text} +

+ ) : null} +
+ ) : null} +
+
+ ); +}; diff --git a/snippets/drama-3/adrian-german.mp3 b/snippets/drama-3/adrian-german.mp3 new file mode 100644 index 00000000..e912097b Binary files /dev/null and b/snippets/drama-3/adrian-german.mp3 differ diff --git a/snippets/drama-3/adrian-japanese.mp3 b/snippets/drama-3/adrian-japanese.mp3 new file mode 100644 index 00000000..b5a6dd90 Binary files /dev/null and b/snippets/drama-3/adrian-japanese.mp3 differ diff --git a/snippets/drama-3/delia-chinese.mp3 b/snippets/drama-3/delia-chinese.mp3 new file mode 100644 index 00000000..5dabdea3 Binary files /dev/null and b/snippets/drama-3/delia-chinese.mp3 differ diff --git a/snippets/drama-3/delia-spanish.mp3 b/snippets/drama-3/delia-spanish.mp3 new file mode 100644 index 00000000..c223ba62 Binary files /dev/null and b/snippets/drama-3/delia-spanish.mp3 differ diff --git a/snippets/drama-3/sheila-01.mp3 b/snippets/drama-3/sheila-01.mp3 new file mode 100644 index 00000000..b2441e20 Binary files /dev/null and b/snippets/drama-3/sheila-01.mp3 differ diff --git a/snippets/drama-3/sheila-02.mp3 b/snippets/drama-3/sheila-02.mp3 new file mode 100644 index 00000000..66e98b14 Binary files /dev/null and b/snippets/drama-3/sheila-02.mp3 differ diff --git a/snippets/drama-3/sheila-03.mp3 b/snippets/drama-3/sheila-03.mp3 new file mode 100644 index 00000000..932460e8 Binary files /dev/null and b/snippets/drama-3/sheila-03.mp3 differ diff --git a/snippets/drama-3/sheila-04.mp3 b/snippets/drama-3/sheila-04.mp3 new file mode 100644 index 00000000..0e35ae4d Binary files /dev/null and b/snippets/drama-3/sheila-04.mp3 differ diff --git a/snippets/drama-3/sheila-french.mp3 b/snippets/drama-3/sheila-french.mp3 new file mode 100644 index 00000000..a33371fb Binary files /dev/null and b/snippets/drama-3/sheila-french.mp3 differ diff --git a/snippets/drama-3/sheila-italian.mp3 b/snippets/drama-3/sheila-italian.mp3 new file mode 100644 index 00000000..c77749f9 Binary files /dev/null and b/snippets/drama-3/sheila-italian.mp3 differ diff --git a/snippets/drama-3/sheila-korean.mp3 b/snippets/drama-3/sheila-korean.mp3 new file mode 100644 index 00000000..c276605f Binary files /dev/null and b/snippets/drama-3/sheila-korean.mp3 differ diff --git a/snippets/drama-3/sheila-phoneme-first.mp3 b/snippets/drama-3/sheila-phoneme-first.mp3 new file mode 100644 index 00000000..02182aa6 Binary files /dev/null and b/snippets/drama-3/sheila-phoneme-first.mp3 differ diff --git a/snippets/drama-3/sheila-phoneme-last.mp3 b/snippets/drama-3/sheila-phoneme-last.mp3 new file mode 100644 index 00000000..c025d4b0 Binary files /dev/null and b/snippets/drama-3/sheila-phoneme-last.mp3 differ diff --git a/snippets/drama-3/ship.mp3 b/snippets/drama-3/ship.mp3 new file mode 100644 index 00000000..4b63a76c Binary files /dev/null and b/snippets/drama-3/ship.mp3 differ