Distillation Using Synthetic Data
topic on 1 show · 1 statements across 1 episodes
1 statements about Distillation Using Synthetic Data, every show
Agarwal: Synthetic data distillation bypasses model vocabulary and tokenizer mismatches
“The one nice thing about this kind of distillation is it doesn't matter if you have a vocabulary mismatch, because we're not using the next token distribution or probability labels. You can distill from one model which uses some random tokenizer to another mod…”