Jun 27, 2024 · 34m · no-priors
No Priors Ep. 70 | With Cartesia Co-Founders Karan Goel & Albert Gu
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of No Priors, Cartesia co-founders Karan Goel and Albert Gu discuss their pioneering work on State Space Models like S4 and Mamba, their mission to challenge Transformer dominance, and the launch of Sonic, their ultra-low-latency voice synthesis engine.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 13.9% of the talking time here. How this is scored →
The hosts as informed peer 4.9 Guest teaching 4.1 Guest disagreement 1.7 The hosts pushing back 1.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 22:14 Karan refutes that speech generation is solved
Karan directly rejects Sarah's premise that audio generation was solved over the past year, asserting existing systems fail basic engagement tests.
Hardest push from the hosts ▶ 12:14 Elad interrogates the DNA modeling claimElad refuses to accept the vague premise of DNA foundation models without understanding whether it targets protein translation or folding.
Biggest teaching moment ▶ 7:55 Albert details why transformers fail on continuous waveformsAlbert explains why throwing transformers at raw continuous signals fails without artificial tokenization, demonstrating the theoretical necessity of SSMs.
The host holds their own ▶ 12:30 Elad demonstrates biology domain depthElad invokes his former career as a biologist to drill down on molecular mechanisms, leaving Albert to admit his own limitations in biology.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Academic Roots at Stanford and the Birth of S4 | 3 | 2 | 1 | 0 | Sarah and Elad invite the founders to recount their academic history at Stanford under Chris Re. The co-founders share lighthearted stories of working together on S4 and filling up Google Cloud disk space. | |
| State Space Models versus Transformer Architectures | 4 | 7 | 2 | 1 | Albert explains state-space models as fuzzy compressors versus transformers. He educates listeners on how transformers struggle with raw waveforms and continuous signals without heavy tokenization. | |
| Linear Scaling, Exact Retrieval, and Hybrid Architectures | 5 | 6 | 1 | 1 | Sarah asks about efficiency versus quality across data types. Albert describes linear scaling advantages and why hybrid architectures with a roughly 10-to-1 SSM-to-attention ratio outperform either architecture alone. | |
| Exploring Domain Applications: From Text to Genomics | 7 | 4 | 2 | 5 | When Albert mentions applying Mamba to DNA modeling, Elad intervenes using his biology background to question what the specific problem formulation is. Albert concedes he is not a biologist and Karan steers the discussion toward audio and edge inference. | |
| The Industry Shift Toward Efficient Local AI Hardware | 6 | 4 | 2 | 1 | Elad and Sarah discuss Apple's 3B on-device models and compute economics. Karan counters that 3B models remain underpowered, arguing SSMs enable true high-performance intelligence on local commodity chips. | |
| The Nuance of Speech: Why Text-to-Speech Is Unsolved | 5 | 6 | 4 | 4 | Sarah pushes back on whether text-to-speech is already solved. Karan forcefully disagrees, outlining the lack of true conversational engagement, intonation nuance, and social role modeling in existing TTS systems. | |
| Unified Multimodal Models versus Pipeline Orchestration | 6 | 4 | 1 | 1 | Elad highlights the severe latency bottlenecks caused by orchestrating multi-model speech-to-text-to-speech pipelines, calling it inelegant. Karan agrees and details Cartesia's strategy to build a unified native multimodal model. | |
| Research Aesthetics and 'Proofs from The Book' | 5 | 3 | 1 | 2 | Albert explains his aesthetic research philosophy, prompting Elad to bring up Erdos's 'Proofs from The Book'. Sarah pushes for a live, un-cooked demo to test latency on a randomized quote. | |
| Team Growth, Intern Culture, and Open Roles | 3 | 1 | 1 | 2 | Sarah jokingly ribs Karan about his large intern ratio. The founders talk through hiring needs for their modeling team and their ongoing mission to overthrow the transformer empire. |
Statements from this episode (16)
Goel: Cartesia's Sonic TTS engine reduces typical voice latency by 150ms
“Even with what we've done with Sonic, we're already kind of shaving off, like, a 150 milliseconds off of you know, what they typically use.”
Goel: Cartesia aims to cut another 600ms of voice latency this year
“And so, you know, the roadmap is let's, let's get to the next 600 milliseconds and try to shape those off in the, over the course of the year.”
Gu: Mamba successfully applied state-space models to language modeling
“Recently proposed a model called Mamba which was kind of brought these to language modeling and showed really good results there.”
Gu: State Space Models can be applied to almost all data types
“So it really can be applied to pretty much everything. So just like kind of Transformers, these are applied to everything. So can these sort of models over the course of research over a few years, we kind of realized that there are different advantages for dif…”
Gu: Early SSMs excelled at raw signals but lagged Transformers on text
“The first types of models we were looking at were really good actually at modeling kind of these raw waveforms raw pixels, things like that, but not as good at modeling text, and transformers are way better there.”
Gu: Transformers struggle significantly on raw pixel or audio waveform data
“People think that like you can throw a transformer at like anything and it just works. Actually it doesn't really like if you try to throw it at like the raw pixel level or the raw sample level and in audio waveforms I think it doesn't work nearly as well.”
Gu: Optimal hybrid models use a 10:1 ratio of SSM to attention
“People have found that the optimal ratio tends to be mostly SSM layers with a little bit of attention. So maybe a ratio of like 10 to one, I know of at least Probably like five groups that have independently verified that this is kind of the optimal ratio of t…”
Gu: Researchers are applying Mamba-based foundation models to DNA sequences
“So I actually just heard from some collaborators today that they applied a mama based model on DNA modeling. They're basically bringing this idea of foundation models To DNA, which is kind of this new idea.”
Goel: AI models generating real-time data will replace traditional graphics engines
“In the future, like you will be essentially replacing, you know, graphics and rendering with essentially models that are outputting streams of data. In real time.”
Goel: Cartesia aims to build edge-oriented infrastructure for SSMs
“I think in general for what we're trying to build is, is sort of the infrastructure to be able to train these models, make them run fast, and then bring them closer and closer to kind of be, you know, very edge oriented rather than cloud oriented.”
Goel: 3B parameter models are still too small and lack strong capabilities
“So the three V models are interesting, I think, but they aren't like, they're still small and not very capable.”
Goel: Current AI speech models fail to capture profession-specific vocal nuances
“And that's sort of the nuance that I don't think any models really capture well, which is like, you know, if you're a nurse, you need to talk in a different way than if you're a lawyer, or if you're a judge, or if you're a venture capitalist, you know, very di…”
Gu: High-quality speech synthesis requires multimodal foundation models
“And so actually to really get, like, perfect even just TTS or, like, speech-to-speech you actually really need to have, like, a model that has, More understanding, like at least of the language, but kind of like, it's not really an isolated component anymore. …”
Goel: Cartesia intends to train an on-device multimodal model
“Yes. But, you know, we have our own sort of set of techniques that we're developing in order to be able to do that effectively. I think that I will maybe leave for another podcast. But I think, yeah, I think that is the intention at the end of the day is build…”
Gu: Aesthetic elegance was the primary driver behind inventing State Space Models
“People ask me, like, how do I treat my research problems, and my, I can't explain. My answer is just aesthetic. It's just like, there's something that I find elegant, and we're aesthetically pleasing about things, and to me, that's almost the most important th…”
Goel: Cartesia runs Sonic TTS model locally on a standard Mac
“Yeah, I have a you know, our model running on our standard issue Mac here. Basically, this is you know, our text-to-speech model. Sonic on our playground is running in the cloud, and so you know, part of what I talked about earlier was how do you kind of bring…”