Larger Models
topic on 4 shows · 6 statements across 5 episodes
Masters of Scale 
Latent Space 
No Priors 
All-In
6 statements about Larger Models, every show
Section 32 does not plan to invest in large foundation models
“I don't plan on investing in, kind of, larger models, right?”
Ben Allal: Small models make more sense than large models for text extraction
“So I think text extraction is like one use case where small models can be really performant, and it makes sense to use them instead of just using larger models.”
Mensch: Advancing reasoning capabilities requires scaling to larger models
“There's still a limit to what a certain model size can do. This limit was, I think, underestimated. But if you want to get to more reasoning capabilities, you do need to move into larger models.”
Mensch: Distilling state-of-the-art small models requires training massive models first
“The other thing about moving into larger models is that it enables you to train smaller models that are better, which is through variety of techniques like distillation or synthetic data generation. So this, these two things are quite related. If you want to m…”
Sutskever: Larger AI models will unlock unprecedented value over small models
“I do think though that as models continue to get larger and better, then they will unlock new and unprecedentedly valuable applications. So yeah, the small models will have their niche for the less interesting applications, which are still very useful.”
Hoffman: Larger AI Models Are Harder to Jailbreak and Deceive
“The larger models are much more easily trainable to say when someone asks for, I'd like to break into the following computer, right? Help me do it. It goes, well, I'm sorry, I can't do that. Today's models, you go, well, my grandmother used to put me to sleep …”