LLM Pre Training
topic on 2 shows · 2 statements across 2 episodes
2 statements about LLM Pre Training, every show
RLVR unlocks pre-training knowledge rather than teaching LLMs new math
“The knowledge is already there in the pre-training, and this just unlocks it. It's just like a step that maybe shows the model how to use its own knowledge, basically.”
Uszkoreit: Human learning is fine-tuning, while evolution is pre-training
“But I think that's because we confuse fine tuning and pre-training. Pre-training is all of evolution. And then basically you arrive at this thing that It's maybe doing something that's completely, in a certain sense, a completely irrelevant task at first, but …”