Disclosure
Kumar: Sesame is open sourcing its speech model, not the full demo
“We're not open sourcing the demo. We're open sourcing the speech generation model that is powering the voice of the demo.”
Assertion Supported
Kumar: Sesame achieves voice cloning via in-context learning prompt strings
“The model is this kind of, you know, it has kind of this in context learning style voice cloning. I mean, typically with some other kind of text-to-speech models, the voice cloning is kind of like an explicit feature. So it's sort of the model has dedicated ki…”
Assertion Contradicted
Kumar: No other open-source model generates multi-participant contextual audio
“At least to our knowledge, there's not another model out there that, that is open source that kind of is a sort of contextual thing where you kind of can put two participants in a conversation, even more, three, and generate kind of a conversation between them…”
Insight
Kumar: Traditional text-to-speech sounds flat because non-neutral tones risk sounding inappropriate
“And that's probably why, or it's one of the reasons why historically voice assistants feel so flat is that traditional text of speech, it's kind of like it can only be flat. Or in other words, if it tries to not be flat, it's very likely wrong.”
Prediction Not checkable as stated
Kumar: Speech research community will shift toward contextual AI architectures
“So, so the speech generation research community is very likely, I think, to move to more and more contextual architectures basically.”
Insight
Kumar: Seemingly easy side products like APIs create massive engineering drag
“Sometimes it feels like an API or something like that is like relatively easy to do. And, you know, it's not like, it's not maybe as hard as some of the other things that we're doing, but everything is a drag on engineering, right?”
Prediction Held up
Kumar: Sesame will not build a one-size-fits-all AI companion
“So we're certainly not going to, we don't see our product as like one companion that's the same for everyone. People have different preferences and that has to be a part of this kind of product category for sure.”
Insight
Kumar: AI conversation is a distinct modality requiring core research
“I think that conversation, like human conversation, is kind of its own modality. And it is nowhere near done, right? There's so much more to do in the core research side to make it better.”
Insight
Kumar: True AI naturalness requires modeling turn-taking and backchannels
“I think to get these things to feel very, very natural and real, you do need to model the full conversation, the turn taking, the back channels, everything.”
Insight
Kumar: Early AI startups need flexible systems thinkers over niche specialists
“Especially when you're smaller, you know, you don't really want to harden, like, you know, you have this team that is super, super niche and doing only this thing, because you don't know exactly what the stack is going to look like tomorrow. Things change on t…”
Assertion Not checkable as stated
Kumar: Word error rate metrics for AI speech generation are now saturated
“Earlier on in the speech generation world in the community, very often you'd look at like word error rate where you look at transcription, like you kind of have a sentence and you generate and you transcribe it and you see if it's the same. And those metrics a…”
Insight
Kumar: Achieving human realism in voice AI is harder than text
“I think you, I think it's much easier. It would be much easier to make a system that produces text chats with you that feels like you're texting a human because there's such a compression of like what the entity on the other side is into just like text. Wherea…”
Insight
Kumar: Adding generative modalities to AI models is harder than understanding
“It's much harder to add a modality to a pre-trained model than it is, add a generative modality, than it is to add an understanding modality.”
Prediction Not checkable as stated
Kumar: Conversational AI will eventually rely on single models over heuristic pipelines
“I don't think you want to, in the long term, have those dynamics be like heuristics and so on, which they kind of are now. There are models involved in some heuristics and so forth. I think in the long term, it's just one model that is kind of naturally employ…”
Disclosure
Kumar: Sesame is developing diffusion-based audio generation models
“We are also working, by the way, on kind of ideas that make the audio generation part diffusion.”
Prediction Not checkable as stated
Kumar: Transformers will remain the dominant AI sequence architecture short-term
“And I wouldn't bet against transformers, you know, not in the short term anyways.”
Disclosure
Kumar: Sesame intentionally includes speech imperfections to make AI sound natural
“My and Miles, they might sort of say the wrong thing or kind of like back up a little bit and say something else or something. And that's on purpose, of course.”
Prediction Not checkable as stated
Kumar: Sesame will preserve AI companion personality as models improve
“They're making assistance. They're making utilities. I love those products. I use them all the time. They're great products. We want to make a companion. And so our prioritization of features and of, let's say, post training kind of personality, et cetera, wil…”
Prediction Not checkable as stated
Kumar: Big tech companies will attempt to own conversational AI interface layer
“I think that over time, I think we will see more of these, you know, bigger companies trying to operate this layer. Like I said, I think that there is not enough effort on that right now, making these systems delightful to interact with, you know, and I think …”
Disclosure
Kumar: The ChatGPT plugin system I built failed to fully take off
“I did this chat to be plugin system before, and I think it's still there probably. And it kind of didn't fully take off really.”
Prediction Not checkable as stated
Kumar: Developer plugins will be essential to future AI interfaces
“I think that the models still need to get better basically to utilize plugins essentially in a way that's kind of reliable enough that someone will go out and look for a plugin for, you know, their kind of downstream service of choice because they just want, y…”
Disclosure
Kumar: Sesame's current demo cannot detect user emotional tone
“The current demo does not sort of hear the user from the perspective of their paralinguistic kind of emotional tone and so forth.”
Disclosure
Kumar: Sesame's entire software and ML team is under 15 people
“The full software team today is still under 15 people, and so we just don't, that's including ML and infrastructure and everything.”
Disclosure
Kumar: Sesame is not pre-training frontier LLMs at scale
“You know, we are not A frontier model company. We're not pre-training LLMs at insane scale and so forth.”
