Self Supervised Learning
topic on 7 shows · 10 statements across 8 episodes
Latent Space 
No Priors 
WTF is with Nikhil Kamath 
the MAD Podcast 
the a16z Podcast 
Big Technology 
20VC
10 statements about Self Supervised Learning, every show
Morcos: Modern AI capabilities depended entirely on self-supervised learning
“But I do not think there's any way we could get to where we are today without self-supervised learning and the ability to train on unlabeled data. That was the real advance to my mind that enabled us to get these incredible increases in capabilities.”
Self-supervised learning is the primary breakthrough behind modern chatbots
“Self-supervised learning is what has become
Very prominent over the last five, six years and is, is really the main component or the main contribution to the success of things like chatbot and natural language understanding systems.”
LLMs are self-supervised models constrained to predict preceding words only
“Chatbots are or LLMs, large language models, are a special case of that, where you train a system to predict a word,
But you only allow it to look at the words that precede it, you know, that are to the left of it.”
Bowen Liu: Small Molecules Currently Lack Self-Supervised Learning Frameworks
“Because like with protein sequences and biological sequences, like you can actually do self-supervised learning, right? Because there is this like complicated generative process with like evolutionary pressure that you can learn from. There's no equivalent for…”
Qiu: Self-supervised AI may learn representations akin to human cognition
“There's something really interesting here where maybe machines are learning the same kinds of representations or similar representations to what humans are learning. And maybe they can get to a point where they can actually do the types of things that humans a…”
Kanjun Qiu: Self-supervised learning mirrors how humans naturally learn
“Self-supervised learning is interesting because it's how humans learn, you know, most of the time we go around We like Google stuff. We learn on our own. We're not giving supervised labels all the time.”
LeCun: Transformer scaling with self-supervised learning surprised AI researchers
“The fact that self-supervised learning methods applied to transformer architectures Work amazingly well, and they work, you know, way beyond what we could have expected. The fact that we can do basically train systems to understand language, translate language…”
LeCun: Joint-embedding self-supervised learning is AI's best shot for world models
“I'm really excited about this area of research because I think it's the germ. I think it's our best shot as to you know, a path towards learning, you know, getting machines to learn abstract representations you know, and more predictive models. Where those pre…”
LeCun: Self-Supervised Learning Forms the Bulk of Natural and Machine Intelligence
“Most of what we learn as humans and animals and in, in the future that machine will learn is learned in this kind of self-supervised manner, basically by watching the world go by and by, you know, taking an action once in a while, but in a, you know, non-task …”
Pesenti: Self-supervised learning will supersede other AI techniques across fields
“Self-supervised learning is really showing promises everywhere, you know, so it's obviously that's the technique behind large language models, but our, you know, my team came up with new papers around vision. So we believe actually that these techniques will s…”