Gradient Descent
topic on 3 shows · 5 statements across 5 episodes
the Y Combinator Startup Podcast 
Latent Space 
No Priors
5 statements about Gradient Descent, every show
Chollet: Gradient Descent Fails at Reasoning by Defaulting to Pattern Matching
“You could not really get Gradient descent to encode sort of like reasoning style algorithms. It was not because the models could not represent these algorithms. It was because gradient descent could not find them, right? So the problem was that it wasn't about…”
Yi Tay: Gradient descent learning paradigm is AI's bottleneck, not architecture
“It's not architecture itself. That's, that there's a problem that we, that is more of like the learning paradigm itself rather than the architecture itself. I think the architecture is just basically like the interface between the learning algorithm and the to…”
Chollet: Gradient descent requires 3 to 4 orders of magnitude more data than humans
“Gradient descent requires vast amounts of data to distill simple abstractions. Many orders of magnitude more data than what humans need. Roughly three to four orders of magnitude more.”
Yao: Reflexion replaces scalar RL rewards with verbal gradient descent
“I think one way to think of reflection is that the traditional idea of reinforcement learning is you have a scalar reward, and then you somehow back propagate the signal of the scalar reward. To the rest of your neural network through whatever algorithm, like …”