distillation using synthetic data
1 statements across 1 episodes · 1 bullish · 0 bearish · 1 people on the record · first statement Mar 23, 2025 by Rishabh Agarwal · across every show →
Everything said about distillation using synthetic data, oldest first
Mar 23, 2025 positive
Agarwal: Synthetic data distillation bypasses model vocabulary and tokenizer mismatches
“The one nice thing about this kind of distillation is it doesn't matter if you have a vocabulary mismatch, because we're not using the next token distribution or probability labels. You can distill from one model which uses some random tokenizer to another mod…”