Proposal: Indic code-switched data (Hindi-English / Tamil-English) + code-switch metric

[balaguhanesh]

#175 adds Hindi data, but monolingual. In production Indian-market voice agents, callers code-switch intra-sentence ("mera appointment reschedule karna hai next Tuesday ko") — exactly where ASR degrades hardest. The ServiceNow code-switching benchmark covers ES/FR/DE-English; no Indic pairs exist anywhere.

I'd like to contribute, in stages:

1. Hindi-English + Tamil-English code-switched utterance sets across existing domains, following Adding Korean and Hindi data #175's conventions (I'm a native speaker of both; I build STT/TTS eval harnesses for code-switching at 2care.ai)
2. Code-Switch F1 scoring alongside WER — accuracy at switch-point words, so boundary degradation isn't averaged away
3. (Follow-up PR) a code-switch perturbation for the perturbation pipeline

Before building: is this something you'd take, and is anyone working on it internally? If green-lit, first set + scoring ready for review in ~a week.

[fanny-riols]

Hi @balaguhanesh , thank you for your interest! And you are right, code-switching is important. We are currently extending our code-switching benchmark to Mandarin, and we were planning to extend EVA-Bench to enable code-switching as well, once we have all the monolingual languages that ServiceNow currently focuses on. But we haven't started on EVA-Bench yet.

We absolutely welcome external contributions. So please go ahead with the data first, and we'll get our team to review. This task would also require some testing of ElevenLabs code-switching capabilities for these languages, as it's the provider we use for the user simulator.