Kumar: Sesame will build a unified audio-text transformer within months
Ankit Kumar · Building the Next Generation of Conversational AI · Mar 15, 2025 · at 1:00:20
Ankit Kumar, cofounder and CTO of Sesame, outlines the company's near-term research roadmap for conversational AI during a podcast with a16z General Partner Anjney Midha.
0:00 / 0:10exact quote · 10.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The path that we're going to take, I think, over the next few months is making a single transformer that does both audio understanding, content, text content generation, and speech generation.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ankit Kumar
Kumar: Voice AI models must decide every 100 milliseconds for natural interaction
“You need to make decisions at the hundred millisecond, let's say, Time segment so that if you're talking and the other person, you know, starts sort of making some noises that make it seem like they're trying to interrupt you or they want to say something, you…”
Kumar: Near-term AI voice models will still lack dynamic conversational fluidity
“And the models that we have today, like CSM, for example, And probably some of the models that we'll have in the short term that will make the experience better will still not be modeling the conversational dynamics because they're making decisions that kind o…”
Kumar: Theoretical gains won't unseat transformers without matching years of optimization
“But because of all the engineering work that the community has done around Transformers, it's like, you know, it's very good. And you're not going to just sort of unseat that, you know, just by an idea, right? There's a lot of work to be done.”
Kumar: Competitors will match Sesame's voice quality; there is no secret sauce
“The other companies, the other sort of chat products and so forth, they will get better voices. They're all, like, it's not gonna, we don't have some magical secret sauce on the technical side that is gonna be, like, impossible to replicate. They're gonna get …”
Kumar: AI voice interface success depends on product experience over model size
“We think that that interface layer, it's really not kind of a core bigger, bigger models, better, better reasoning question. It's really a product experience question. It's really a question of, can you make a system that people actually want to interact with,…”
Kumar: Transcription-free conversational AI models are coming soon
“A pretty clear path that a lot of, I think, labs are taking, and we're taking as well, and will be in kind of future versions, is just kind of transcription free. Just go straight into the text component which will kind of obviate transcription entirely. That …”