Dense Models
topic on 3 shows · 3 statements across 3 episodes
Latent Space 
No Priors 
the MAD Podcast
3 statements about Dense Models, every show
Catanzaro: Dense models outperform MoE models under strict memory constraints
“You know, they take a lot more memory. If you have a very small amount of memory, a dense model is going to be smarter.”
He: Upcycling dense models to MoE beats continuing dense training per FLOP
“By training these upcycled models, you can achieve better accuracy than simply training the dense model further for the same number of flops.”
Guu: AI Researchers Should Prioritize Making Dense Representations More Controllable
“If I were kind of advising a student or something on looking into knowledge representations, at the moment I would say there's a lot of momentum on, on dense models continuing to capture more and more of the different applications, and so if there is a way tha…”