Mosaic MPT
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement May 31, 2024 by Mark Huang · across every show →
Everything said about Mosaic MPT, oldest first
May 31, 2024 neutral
Huang: Modern LLMs have shifted from ALiBi to RoPE scaling
“Some of the newer architectures don't actually employ it a lot. I think the last architecture that actually really employed it was the Mosaic MPT model class, and then almost all the models these days are all rope scaling, and then effectively you can use yarn…”