feat: Implement fallback for audio files - #420
lukasdotcom wants to merge 1 commit into
Conversation
7ba3467 to
7b4d9e0
Compare
|
Were there any models you encountered that fall under this category? And did you notice any difference in output quality between transcript vs. direct audio file? |
Not all models support audio files. For example qwen does not support audio files. I think gemma understands audio files in a similar way anyway as it sometimes acts confused if you ask it to summarize an audio file and says it only sees a transcript. For the topic of quality I don't know if for sure I would guess this is lower quality than an actual understanding, but this is really meant for models that don't support audio files anyway. |
Hm, this doesn't sound very user-friendly. Would it be worthwhile to just prevent this case from happening by e.g. disabling audio file input for unsupported models, or would it be too much work? |
Sorry I think I wasn't clear enough gemma supports audio files natively, and had that problem. This method works fine the model thinks it was actually given an audio file and would respond to understanding what the audio is about (other than for cases where the transcript doesn't give enough information. Eg: what bird is that) |
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
7b4d9e0 to
477fa66
Compare
| } | ||
| return [[ | ||
| 'type' => 'text', | ||
| 'text' => 'Filename:' . $file->getName() . "\nTranscription:\n" . $resultTask->getOutput()['output'], |
There was a problem hiding this comment.
The force push fixed a typo here the \n was missing the n for one of them
2 participants
Fallback for audio files that they get transcribed to text if the model doesn't support audio natively.
🤖 AI (if applicable)