Pre Trained Model
topic on 4 shows · 6 statements across 6 episodes
Latent Space 
No Priors 
the MAD Podcast 
the a16z Podcast
6 statements about Pre Trained Model, every show
Roberts: Powerful pre-trained models are necessary for effective RL and reasoning
“If you have a powerful enough pre-trained model, then it can start to do well at RL. It can start to like think at use test time compute to for instance, solve, solve math problems that it wouldn't otherwise be able to do.”
Schrittwieser: Raw pre-trained AI models make poor agents without RL
“Our pre-training data is not very agent-like. If you think of the pre-training data, right, there is like websites and books and, you know, all kinds of recent text that has a lot of information, but it doesn't have a lot of actions. It doesn't really capture …”
Cubuk: Periodic Labs mid-trains existing LLMs rather than building from scratch
“We take a pre-trained model and then mid-train it, you know, high computer.”
Noam Brown: Models need baseline capabilities to benefit from test-time reasoning
“One thing that I think is underappreciated is that the models, the pre-trained models need a certain level of capability in order to really benefit from this, like, extra thinking.”
Singhal: Fine-tuning outperforms prompt tuning when providing over 100 examples
“If you have three to five examples, let's say, then I would prompt it. If you have maybe 10 or 50 examples, it would either be prompt tuning or fine tuning. I think generally in that realm, prompt tuning and fine tuning perform similarly, and I would prefer pr…”