The Wisdom Wall
13 quotable lessons, heuristics and mental models. Every one is playable at the moment it was said. No fortune cookies allowed.
“What matters in reality is making things work. And it's often the simplest ideas, uh, that work in practice, especially when, uh, thrown a lot of compute at them. The simplest ideas typically outshine the complicated ones.”
“hyper-optimizing for margins now would be the wrong, uh, tactical move.”
“Meta has a bigger moat than Google, uh, because Google's moat on distribution comes from their deals with, with carriers, OEMs, and like all, you know, all these people, um, but Meta's, um, moat is this raw network effects, like, Nobody pre-installs Instagram or WhatsApp on phones. Android pre-installs Google. Despite…”
“every three months or something like there's a new open source model out there, uh, and then that forces them auto labs to lower the prices because then nobody's going to use their APIs.”
“If AI was not neural nets, then GPUs wouldn't have mattered. But, but AI happened to be just basically neural nets at scale. And so all the primitives they built, all the, uh, software stack they built ended up being, like, The, the core foundational building blocks for neural networks too.”
“I think it still helps to be very good at infrastructure, backend, uh, data centers, like, uh, floating point arithmetic, storage, all the core fundamentals are not going away. In fact, like, I would say they're very essential in a world where AIs are taking care of the front end and the UI and, um, all that stuff…”
“physical access matters way less anymore. I think it's more the amount of time you get to spend yourself with an AI model using these apps, understanding where they fail and, uh, talking to the best people.”
“what was really on the frontier of science at that time when I was doing PhD was like, how can we figure out general intelligence? Uh, in a manner similar to a human, which is one system doing hundreds of thousands of tasks without explicitly being programmed for it, and can be taught new tasks, uh, and, and, and, and…”
“If you're training it on the raw stock price, let's say you just have a bunch of numbers of the stock price of Nvidia, uh, opening price every single day. Sure, it's not going to be useful on its own, because there are so many other factors that influence the price, and if, if all it had is like the each day's opening…”
“There are so many other ways to do machine learning that are like, you know, support vector machines, linear regression, logistic regression, there's like a whole bunch of techniques, but it happens to be that, uh, neural networks is the one way to do things when you really want to benefit from scale. Like, like, like…”
“Generalization across different physics settings is still like pretty bad. It's not like training on the internet. There's not enough data. So you actually have to build something that's truly, uh, intelligent so that it can learn with very little data.”
“compute alone is useless. Like, people have tried to reproduce these things with doing the same thing, and it doesn't work. You gotta throw high-quality data tokens at the problem, too.”
“if you want reasoning to emerge in a model, it's good for you to, like, make sure you have YouTube transcripts of video, uh, like, like, like, Uh, lectures, uh, MIT lectures, Stanford lectures, um, and, and textbooks, like where you actually have problems, where it's not just a problem, but the solution is explained…”