pre-training
5 statements across 5 episodes · 3 bullish · 0 bearish · 5 people on the record · first statement Apr 25, 2023 by Dr. Percy Liang · across every show →
Everything said about pre-training, oldest first
Apr 25, 2023 positive
Liang: Controlling AI hallucination is easier once models understand the concept
“So I think there's pre-training, which is predicting the next word and developing a world model, so to speak. And with those capabilities, then you can, you still have to say don't hallucinate, but it will be much easier to control that model if it has a notio…”
Aug 1, 2024 bullish
Vinyals predicts pre-training compute will drop to ~50% as RL expands
“So to me, that balance feels correct, like some on pre-training, and here we, we're trying to learn every task. So certainly that's going to be, you know, let's say it can be as high as 50%, not as high as over 90 like today. And then the rest mostly on reinfo…”
Sep 19, 2024 neutral
Taylor: AI pre-training will consolidate into a small number of frontier model builders
“I think that will probably play out with the frontier models. We'll end up with a relatively small number of companies doing pre-training you know, which is the really capital intensive part of model building not because, you know, there's not, they're the onl…”
Nov 21, 2024
Jul 17, 2025 bullish
Laskin: RL requires far fewer FLOPs than pre-training for frontier models
“We're in this brief period in history right now where the RL flops are still manageable. Like you can really have a best in class product if you're focused. And yes, you'll need to put, you know, you still need a decent amount of GPUs, but from a flops perspec…”