Mascorro: DeepSeek-R1 post-training used two SFT and two RL phases
Marco Mascorro · DeepSeek, Reasoning Models, and the Future of LLMs · Mar 5, 2025 · at 11:25
Marco Mascorro, partner at a16z, breaks down the technical post-training steps used by AI lab DeepSeek to convert DeepSeek-R1-Zero into the user-friendly DeepSeek-R1.
0:00 / 0:13exact quote · 13.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So basically the way they did that, trying to fix R one zero, is it added a couple more phases in the post-training. That included two supervised fine tuning phases and two reinforcement learning phases. And these reinforcement learning phases, they were a large scale reinforcement learning phases.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Marco Mascorro
Mascorro: DeepSeek-R1 proved reinforcement learning improves models without human feedback
“And I think the big thing in, in R-one, or generally with these reasoning models is, We were doing before there was a human in the loop always, right? Like when we have this SFT training and these other techniques that we're doing after like RLHF and having R …”
Mascorro: Distillations from DeepSeek-R1 Outperformed Direct RL on Smaller Models
“So it turns out in their experiments, they took Lama's EV and some of these are QN models, and they basically apply RL straight the same way they did it with R one on these base models. And it turns out that it improved in some fields, but it was not a signifi…”
Mascorro: High-quality LLMs can be built purely with SFT data
“Now, the reality is like you can get to really good models purely with like SFT data.”
Mascorro: DeepSeek consistently open-sources its model weights and training techniques
“So, one of the good things about DeepSeek is basically they open source their weights, their techniques, and how they build these models, and they've been doing that for a while.”
Mascorro: DeepSeek-R1-Zero Improved Math Scores but Struggled with Readability and Language Switching
“R one zero, which in a way was a very interesting model because it showed that it improved in some reasoning benchmarks and math benchmarks. But eventually didn't do really well on other things, right? Like it was switching between languages. I think that was …”