Smaller Models
topic on 7 shows · 16 statements across 15 episodes
the Y Combinator Startup Podcast 
Masters of Scale 
Latent Space 
No Priors 
the MAD Podcast 
the a16z Podcast 
20VC
16 statements about Smaller Models, every show
Kant: Industry will squeeze far more capability from smaller models via behavior
“We are going to be able to squeeze so much more out of smaller models than I think we had imagined in the industry. Because yes, there's intelligence and larger models are more intelligent. Like no doubt about it. We should continue to scale up. But the behavi…”
Krishna: Enterprises will transition to specialized smaller AI models within two years
“And this is the transition that I'll predict will happen within 24 months. I'm not sure it'll happen within 12 months.”
Jeff Dean: Capable small models require first building frontier models
“Through distillation, which is a key technique for making the smaller models more capable, you know, you have to have the frontier model in order to then distill it into your smaller model. So it's not like an either or choice. You sort of need that in order t…”
Dettmers: Frontier AI performance will stagnate while smaller models improve
“Performance on the frontier will stagnate, but on the smaller level, we get more and more powerful models still, because you can distill from these large models into these small models.”
Andreessen: AI market will split into supercomputers and small embedded models
“I tend to think the AI industry is going to be structured a lot like the computer industry ended up getting structured, which is you're going to have a small handful of basically the equivalent of supercomputers, which are these like giant, you know, kind of w…”
Morcos: Test-time compute fundamentally favors smaller models to cut multi-step inference costs
“Test time compute as a paradigm really pushes you towards smaller models, right? Because if your cost of solving a problem is cost of inference times number of thinking steps, and you have to do a lot of thinking steps. Well, now this is like a really like min…”
Morris: Two years of academic AI research on small models was inconsequential
“There was like kind of two years where everyone in academia was working on like smaller models and none of it really mattered.”
Morin: Distilled smaller AI models can outperform their larger base models
“Probably the most, I would say mind blowing thing about distillation is that sometimes the smaller models become better than the bigger model through distillation.”
Morin: AI developers will always choose smaller models if performance matches
“What really pushes model sizes are the efficiency rather than specializing. So meaning that if you can do the same performance with a smaller model that is fine tuned with rag or whatever, then you'll do it with a smaller because again, less is better.”
Jamil: Smaller models gain more improvement from Writing in the Margins
“As you can see, for example smaller models have a better more more improvement.”
Argenti: Enterprise AI will combine proprietary reasoning with specialized internal models
“And I think the same is going to be with AI, where you're going to have a reliance on, You know, large proprietary models, mostly for the reasoning capabilities, almost like, you know, I really want to understand what you're asking. And then kind of they will …”
Bach: Smaller, more powerful models will ensure unconstrained AI remains accessible
“Yes, but there will also be better jailbroken models or models that have never been jailed before, because we find out how to make smaller models that are more powerful.”
Sutskever: Reliability is the biggest bottleneck to truly useful AI models
“I would actually point out that the main thing that's lost when you switch to the smaller models is reliability. I would argue that at this point it is reliability that's the biggest bottleneck to these models being truly useful.”
The AI market will shift toward smaller models running without GPUs
“It's one of the reasons I think that we're going to see more smaller models and you've got startups now emerging, which are essentially trying to, like, distill down, like from a large model into something much smaller that will run without a GPU, and they're …”
Appenzeller: Over-training smaller AI models allows them to match larger ones
“You can match the performance of a large model with a smaller model if you train it more, right?”