Data Efficiency
topic on 2 shows · 4 statements across 3 episodes
American Optimist 
Latent Space
4 statements about Data Efficiency, every show
Yi Tay: Data-Bound AI Must Spend More Compute Per Token to Scale
“So if you are, you come to a point where you are Very data bound, but not compute bound at all. You just find algorithms that spend a lot of compute on every token.”
Nair: AI models lag orders of magnitude behind human one-shot error learning
“It seems like we're kind of, like, a few orders of magnitude of, like, kind of data efficiency, basically, away from, like, that kind of, like, you know, you do something once, or, like, you make a mistake like, you, yeah, you introduce, like, a bug in your co…”
Nair: Continual learning breakthroughs will be paradigm-shifting within a year
“I suspect that it will be kind of, like, paradigm shifting in the next, like, year or something, but I have no idea, like, you know, what it might be.”