Supervised Fine Tuning
topic on 3 shows · 5 statements across 5 episodes
Latent Space 
the MAD Podcast 
the a16z Podcast
5 statements about Supervised Fine Tuning, every show
Dubois: Supervised fine-tuning cannot exceed human demonstrator capabilities
“The problem with this is that you will never get better than what your ground truth gives you. And humans are actually pretty limited in many sense, so you will never, like, overcome the human labelers that you, you're working with.”
Yi Tay: On-policy RL is more generalizable than imitation fine-tuning
“So I think on policyness is basically this idea of like model training on its own outputs and letting the model like generate its own trajectories and then letting some reward verify it and then the model train its own outputs. I think this is more generalizab…”
Sherman Wu: OpenAI RFT elevates domain models to state-of-the-art performance
“The reinforcement fine tuning API... Kind of changes the paradigm away from just, like, small incremental, like, tone improvements, which is what SFT did, to actually improving the model to potentially SOTA level on a particular use case that you know about.”
Jin: Policy gradient algorithms function as weighted supervised fine-tuning
“If you kind of, like, look at, if you kind of stare at, like, this part it sort of looks like just, like, weighted supervised fine-tuning, right? Like, you have this, like, log of, like, the probability of a token and, like, some, like, weight on it and reinfo…”
Reinforcement Learning Fails Without Initial SFT to Seed Rewardable Behaviors
“It can potentially work otherwise, but practically it only works when the agent has interacted with a reward, right? It's received a positive reward for what it's done. Maybe one out of 10 times, one out of 50 times, but if it's getting zero reward, then you d…”