pre-training

“You need knowledge in order to build reasoning on top of it. Right. a model can't kind of go in blind and just learn reasoning from scratch. So we find these two paradigms to be fairly complementary and we think, you know, they have feedback loops on each oth…”

“So you don't want to just filter out everything so that the model doesn't know anything about it but at the same time, you don't want it to output, you know, how to build a bomb so there's like a fine balance here, and that's why pre-training is defined as pre…”

“Pre-training, which is this idea that you just make the model bigger that has had diminishing returns.”

“And this is a strength, a sound strategy because you have everybody talking about how pre-training is hitting diminishing returns. You have everybody talking about how data is hitting a wall. And so what do you need? You just need these algorithmic development…”

“It does seem like post training is where the best innovations are happening now and the pre-training and the amount of data, like they've, we've used up a lot of the data. They're trying to create synthetic data to try to improve model performance.”

“Every single step of the model production pipeline multiplies. And so you want to improve all of them. And the thing that we see is we prove the pre-training. It makes all the other steps much easier. And it makes sense because it's a model is able to learn fa…”

“Once investors look under the hood and see more and more of this, they'll be questioning, why are we spending so much on pre-training? Why are you doing billion dollar training runs anymore? If most of the gains and models are coming from post training and RLH…”