Jeffrey: Canada hosts three of the world's top AI institutes
“Three of the top AI institutes in the world are in Canada. One's in Montreal. It's called Mila, around Yoshua Bengio who has invented deep neural nets, as well as the one in Toronto around the Vector Institute, which is Jeff Hinton, who is a Nobel Prize laurea…”
Pre-training models on language before reinforcement learning is the correct architecture
“Having the model have a prior of language and being able to like, think in language and then train on top of that, that seems like clearly the right. The right thing to do.”
Dan Roberts: AI scaling requires novel algorithms, not just pure compute
“It's not that scale is all you need. You need to also have good ideas to guide the scaling.”
Lambert: Rich Sutton's RL theories are impractical for models like GPT-6
“Rich is a font of wonderful ideas, but Often not ones that are going to be immediately practical. This is how you get things like creating reinforcement learning, but not necessarily things that are going to impact what GPT six is.”
Rabois: The Bitter Lesson requires data that may take decades to acquire
“Conceptual, you know, the blog post is right, but that's only true when you have enough data, and depending on the use case, the level of data you need may not be possible for years, decades, and you may need to hack your way there through human interactions.”
Schuler: Rich Sutton's textbook is computer science's most influential publication
“According to the Alan Soufre AI Semantics Scholar, he's the most highly cited researcher in reinforcement learning, 11th most influential researcher in all of competing science. And his textbook on reinforcement learning was ranked as the single most influenti…”