The Wisdom Wall

“And like, the problem is super simple to define. It's just like, um, predict the next word, the fat cat sat on the, you know, like, okay, well, you know, what comes next? Like it's, it's, it's extremely easy to define. And if you can do a great job of it, like, you know, then you get everything that you're seeing right…”

“Like if you're like such a public, you know, trying to present like one public persona that everyone likes, you're going to end up just being boring essentially. And people just don't want boring. You know, people want like the, you know, uh, want to interact with, uh, something that feels human, you know?”

“really the key insight is that what makes deep learning work is that it is, um, really well suited to, uh, to modern hardware, um, where, you know, you have the, uh, the current generation of, uh, of chips that are great at, um, uh, Matrix multiplies and, you know, other, other forms of things that require large…”

“probably you don't need that high end, like level of intelligence, like To, you know, to do, to do emotion.”

“an image is worth a thousand words, but it's like a million pixels. So like the text is like a thousand times as dense.”

“the insight here was, Hey, you can use the same attention thing to like, look back at the past of the sequence that you're, that you're trying to produce. And you know, the beauty is that the, that It runs great on, uh, on GPUs and CPUs, and, um, it's kind of parallel to, like, how deep learning is, uh, like, has taken…”

“an image is worth a thousand words, but it's like a million pixels. So like the text is like a thousand times as dense.”

“the data requirements tend to go up like with the square root of the amount of computation, because you're going to train a bigger model and then you're going to, um, throw more data at it.”

“this is kind of a technology that, um, that's so accessible that like billions of people can just invent use cases, you know, and, and like, it's so flexible that, you know, you really just want to put the user in control because often they know way better than you do what, what they want to use the thing for.”

“the way you have a company that's, that's both AGI first and product first is that you make the, make your product depend entirely on the quality of the AI. Like the, the most, the biggest determining factor in the, in the quality of our product is, is how smart things gonna be.”

“the magic of transformer kind of like convolutions is that you get to process the whole sequence, uh, at once. I mean, it, it still talks, you know, it's still a function of like the, you know, the, the predictions for the later words are, uh, dependent on what the earlier words are, but it happens in like a constant…”

“put in a greeting, a name in a greeting is, uh, is all you need typically for, um, you know, for famous, uh, like famous characters cause like, or famous people cause the, um, the model probably already knows, uh, what, uh, what they're supposed to be like.”