I feel like there is a lot of room to build great voice-based agents that don't exist right now.
I have found that ChatGPT voice mode is unusable (e.g. hallucinates me saying things); Claude voice mode is usable, but very buggy around tool calling, and it often mishears things. And it only supports Opus, not Fable (though it looks like you don't support either of those). But I use it anyway.
Question, do any of your TTS options support increasing the speaking speed?
show comments
webo
The benchmarks page seems interesting and something I can use to help make an informed decision. Can you talk about how you're measuring some of these? I imagine it needs to involve some human input.
Does this include a turn taking API? It'd be great to have one API that could do "Conversation in a box". One of the biggest annoyances is daisy chaining many models together for turn taking, dumb models for immediate responses, with smarter models returning and taking over after.
Which model best allows me to transcribe speech that uses a lot of domain-specific terms? For example, when I say "Claude Code", it often gets transcribed as "Cloud Code", and I have to go back and edit or do a second pass with a traditional LLM (which can introduce additional errors).
show comments
Tepix
Ever since OpenAI launched their improved voice mode, I've been looking for a capable LLM with builtin voice-in and -out. Llama 4 was supposed to be it but turned out to be a dud. I haven't followed the topic closely lately, did I miss anything? Are there capable (!) open weights omni models that allow low latency voice chat? If so, what software do you use them with? Can you use a PWA on your phone? WebRTC? WebTransport?
show comments
alexcnwy
why isn't openrouter gonna be the openrouter for voice ?
Seems to be useless, the state of art for all categories is local on-device, voice model vendors are just rent seekers for those who know no better.
show comments
cnqso
What does WER/CER stand for? I see it as some kind of key metric under each model but not clear what it's measuring
show comments
aleksiy123
Is this only for realtime tts use cases?
Wondering if you also support some non realtime models.
show comments
dayvough
Looks awesome, can't wait to try it for some Filipino workflows when it's available!
show comments
dhruv3006
the concept is interesting I must say - good luck !
show comments
IgorBlink
any progress with on-device models??
show comments
narrationbox
> Typical production voice agent is an ensemble of three models: STT, an LLM, and TTS.
To use a claudism, I would like to push back on this. The industry is very much moving towards one-model-does-all end to end trained similar to LLMs and VLMs. Mostly for latency reasons and partially because the results for the end to end trained models are just so much better than those using three pieces architectures.
I think most of the value prop is in automatic evals, not routing specifically. A better pitch for you would be "the LM Arena of voice models" rather than comparing yourself to openrouter because the value add is rather questionable. For TTS specifically, the current SOTA for production systems are all using prompt based voice gen i.e. instead of having 10 different Tacotron models trained on 10 different models, these days it's all a single large model and the "style" is a prompt in the system prompt. The input is usually something like
<System prompt>
Speak in a deep smooth voice similar to a documentary narrator
</System prompt>
<Text to Narrate>
Speko is the ultimate evaluation platform for voice agents. We do automatic evals.
</Text to Narrate>
It's the same for voice cloning too, you just pass the reference speech as an input file for all generations. A lot of systems don't have any separate style vector extraction step or model-specific fine-tuning anymore.
So something like OpenRouter for voices offer questionable value given that stakeholders usually make this sort of decisions once at the start of the project. On the other hand if you can offer automatic evals and figure out which prompts give the most similar results across different voice providers, that would offer a lot more value. It would be nice to be able to switch from e.g. Grok voice agents to ChatGPT voice agents knowing that the output style won't change too much. There are many companies now with evals as a core business model: LM Arena, Artificial Analysis, Prompt foo (before they got acquired and pivoted to security only) so many take a look at them.
This looks really interesting.
I feel like there is a lot of room to build great voice-based agents that don't exist right now.
I have found that ChatGPT voice mode is unusable (e.g. hallucinates me saying things); Claude voice mode is usable, but very buggy around tool calling, and it often mishears things. And it only supports Opus, not Fable (though it looks like you don't support either of those). But I use it anyway.
Question, do any of your TTS options support increasing the speaking speed?
The benchmarks page seems interesting and something I can use to help make an informed decision. Can you talk about how you're measuring some of these? I imagine it needs to involve some human input.
https://benchmarks.speko.ai/turntaking
Is voice the right form factor? If so then search aka google will win in practicality no?
https://s-1.vercel.app/posts/why-openrouter-can-be-the-next-...
Does this include a turn taking API? It'd be great to have one API that could do "Conversation in a box". One of the biggest annoyances is daisy chaining many models together for turn taking, dumb models for immediate responses, with smarter models returning and taking over after.
I made a completely free 100% on device translation app https://apps.apple.com/us/app/arda-translate/id6778970560 and hard to image a world where TTS and STT will not be done locally in the future
Which model best allows me to transcribe speech that uses a lot of domain-specific terms? For example, when I say "Claude Code", it often gets transcribed as "Cloud Code", and I have to go back and edit or do a second pass with a traditional LLM (which can introduce additional errors).
Ever since OpenAI launched their improved voice mode, I've been looking for a capable LLM with builtin voice-in and -out. Llama 4 was supposed to be it but turned out to be a dud. I haven't followed the topic closely lately, did I miss anything? Are there capable (!) open weights omni models that allow low latency voice chat? If so, what software do you use them with? Can you use a PWA on your phone? WebRTC? WebTransport?
why isn't openrouter gonna be the openrouter for voice ?
What is the difference with Livekit Gateway? https://livekit.com/blog/introducing-livekit-inference
Or even something more managed like Vapi?
Seems to be useless, the state of art for all categories is local on-device, voice model vendors are just rent seekers for those who know no better.
What does WER/CER stand for? I see it as some kind of key metric under each model but not clear what it's measuring
Is this only for realtime tts use cases?
Wondering if you also support some non realtime models.
Looks awesome, can't wait to try it for some Filipino workflows when it's available!
the concept is interesting I must say - good luck !
any progress with on-device models??
> Typical production voice agent is an ensemble of three models: STT, an LLM, and TTS.
To use a claudism, I would like to push back on this. The industry is very much moving towards one-model-does-all end to end trained similar to LLMs and VLMs. Mostly for latency reasons and partially because the results for the end to end trained models are just so much better than those using three pieces architectures.
I think most of the value prop is in automatic evals, not routing specifically. A better pitch for you would be "the LM Arena of voice models" rather than comparing yourself to openrouter because the value add is rather questionable. For TTS specifically, the current SOTA for production systems are all using prompt based voice gen i.e. instead of having 10 different Tacotron models trained on 10 different models, these days it's all a single large model and the "style" is a prompt in the system prompt. The input is usually something like
It's the same for voice cloning too, you just pass the reference speech as an input file for all generations. A lot of systems don't have any separate style vector extraction step or model-specific fine-tuning anymore.So something like OpenRouter for voices offer questionable value given that stakeholders usually make this sort of decisions once at the start of the project. On the other hand if you can offer automatic evals and figure out which prompts give the most similar results across different voice providers, that would offer a lot more value. It would be nice to be able to switch from e.g. Grok voice agents to ChatGPT voice agents knowing that the output style won't change too much. There are many companies now with evals as a core business model: LM Arena, Artificial Analysis, Prompt foo (before they got acquired and pivoted to security only) so many take a look at them.
Source: we have been building TTS systems for over a decade too https://narrationbox.com
The link, since it seems to be missing?
https://speko.ai/
cool! we have something similar implemented: https://github.com/fastrepl/anarlog/tree/main/crates/owhispe...