Sesame

“Maya and Miles today, they can't do anything for you”

“At least to our knowledge, there's not another model out there that, that is open source that kind of is a sort of contextual thing where you kind of can put two participants in a conversation, even more, three, and generate kind of a conversation between them…”

“I do think that there is kind of an under-investment or an under-focus in the sort of Strong AI team world on product experience and sort of creative taste and kind of humanities maybe, in a sense, to kind of bring AI to experiences that kind of everyday peopl…”

“You know, we are not A frontier model company. We're not pre-training LLMs at insane scale and so forth.”

“Especially when you're smaller, you know, you don't really want to harden, like, you know, you have this team that is super, super niche and doing only this thing, because you don't know exactly what the stack is going to look like tomorrow. Things change on t…”

“We are open sourcing the speech generation base model basically. And so the base model can generate any voice. It's quite conversational, but you do need to fine tune it probably if you want to get a particular personality or a particular kind of voice out of …”

“The other companies, the other sort of chat products and so forth, they will get better voices. They're all, like, it's not gonna, we don't have some magical secret sauce on the technical side that is gonna be, like, impossible to replicate. They're gonna get …”

“We think that that interface layer, it's really not kind of a core bigger, bigger models, better, better reasoning question. It's really a product experience question. It's really a question of, can you make a system that people actually want to interact with,…”

“And so the kind of next versions of our models that will take audio and natively into the kind of LLM component will hopefully more and more pick up on those things.”

“A pretty clear path that a lot of, I think, labs are taking, and we're taking as well, and will be in kind of future versions, is just kind of transcription free. Just go straight into the text component which will kind of obviate transcription entirely. That …”

“We're not open sourcing the demo. We're open sourcing the speech generation model that is powering the voice of the demo.”

“When you build the thing, right, when you're building the product and using it every day, you know, there are some things that you work on that don't get into the demo because they're going to take longer and you want to ship the demo. You kind of know how big…”

“Misleading at times because you tried so much and you don't have, at least when we're trying it internally, you don't have such a diversity of users that you get kind of the first reaction over and over, right? You only get so many first reactions. And then wh…”

“I think to get these things to feel very, very natural and real, you do need to model the full conversation, the turn taking, the back channels, everything.”

“Like we had to build the models that we're going to open source from scratch in order to get them to a point where they can achieve this experience.”

“But other parts in particular, kind of some of the personality aspects, some of the voice aspects, the speech generation, we didn't think and we still don't think will just be kind of done by the community. We think we will need to do it because that's kind of…”

“Relative to five, six days ago, I don't think it's too much of an exaggeration anymore to say this research preview has been for voice what ChatGPT was for text.”

“Earlier on in the speech generation world in the community, very often you'd look at like word error rate where you look at transcription, like you kind of have a sentence and you generate and you transcribe it and you see if it's the same. And those metrics a…”

“We are making an app. We will make an app. I think for a little bit of time, it's going to still be kind of the demo experience. We want to support people using that for a long time, or, you know, we don't want to, we're not taking it away anytime soon from wh…”

“The current demo does not sort of hear the user from the perspective of their paralinguistic kind of emotional tone and so forth.”

“We mentioned on the website and we mentioned some of our launch content that we are working towards glasses as a form factor for Companions, or kind of this companion interface”

“I think that over time, I think we will see more of these, you know, bigger companies trying to operate this layer. Like I said, I think that there is not enough effort on that right now, making these systems delightful to interact with, you know, and I think …”

“We also have some data sets that are kind of like Just two people in a conversation or sometimes they're actors, but it's trying to be a real conversation. And so we'll kind of take the conditioning of some snippet of the conversation and then show a human rat…”

“And the models that we have today, like CSM, for example, And probably some of the models that we'll have in the short term that will make the experience better will still not be modeling the conversational dynamics because they're making decisions that kind o…”

“We want You know, sub-five hundred millisecond response times, and a lot of things that feel like not a big deal, 50 milliseconds here, 50 milliseconds there, can really add up.”

“So, you know, if you talk to Maya and Miles, you probably will not be able to get the same quality of like reasoning capabilities or intelligence as other As other systems, but in return, you're kind of getting this much more natural fluid interaction.”

“You need to make decisions at the hundred millisecond, let's say, Time segment so that if you're talking and the other person, you know, starts sort of making some noises that make it seem like they're trying to interrupt you or they want to say something, you…”

“They're making assistance. They're making utilities. I love those products. I use them all the time. They're great products. We want to make a companion. And so our prioritization of features and of, let's say, post training kind of personality, et cetera, wil…”

“My and Miles, they might sort of say the wrong thing or kind of like back up a little bit and say something else or something. And that's on purpose, of course.”

“But really, I think with some of these more product experience questions, there's something qualitative about it that is very hard to quantify. That is one of the big challenges internally, actually, is how do you hill climb effectively on what is really an ML…”

“I think you, I think it's much easier. It would be much easier to make a system that produces text chats with you that feels like you're texting a human because there's such a compression of like what the entity on the other side is into just like text. Wherea…”

“The full software team today is still under 15 people, and so we just don't, that's including ML and infrastructure and everything.”

“So we published in the, in our blog post, we trained three variants. We trained 1,000,000,003 1,000,000,008 billion of just speech generation.”

“The model is this kind of, you know, it has kind of this in context learning style voice cloning. I mean, typically with some other kind of text-to-speech models, the voice cloning is kind of like an explicit feature. So it's sort of the model has dedicated ki…”

“The path that we're going to take, I think, over the next few months is making a single transformer that does both audio understanding, content, text content generation, and speech generation.”

“We are also working, by the way, on kind of ideas that make the audio generation part diffusion.”

“And I think that we will see a lot of not just Sesame, but other kinds of media, let's say, that are sort of AI native in a way that Bring some creativity, bring some like storytelling into AI, or maybe bring AI into those categories. And I think they'll make …”

“We are not a developer facing business. We're not making an API.”

“The speech generation part is conditioned on the, on all the audio of the conversation.”

“I think from my perspective, good taste in ML today, because it's such a fast moving field with so many people working across, you know, open source and APIs and big labs and so forth. Really, you're trying to identify What part of the ecosystem or what part o…”

“So we're certainly not going to, we don't see our product as like one companion that's the same for everyone. People have different preferences and that has to be a part of this kind of product category for sure.”