Attention

“Yeah, on Sudoku, MLP actually outperformed the attention, It was it scored zero on the maze, ah, the MLP scored zero on the maze, and so there's, it's not clear, it's not obvious that, ah, the transformer is always better.”

“It goes back to like one of our founding principles, which is attention is more valuable than capital.”

“The mixture of expert layer maps really well, but the attention does not.”

“Most of that is an attention and MOPs, right? So if you have good kernels for attention, MOPs and norms and so on, then it doesn't much matter what the front end to, you know, send tensors to and from those kernels is”

“And while people have found modifications to the transformer, which make the constant time overhead of attention, reduce the constant time overhead on attention to like, you find these little hacks with mixture of experts or latent attention, nobody has gotten…”

“So like, I think that right now we have some pretty good solutions for like understanding what's in the residual stream, understanding what's, is it in MLPs? We don't have good solutions for like attention.”

“I believe that the internet and attention is the most powerful weapon that you have today in the 21st century, and if you have money, you can buy attention but if you don't have money, you can't buy attention”

“One of the things you learn is that attention and attentional flows and interest are these sort of forces that are outside your control. They kind of, like, move in and out like the wind, and then you try to do with them what you can.”

“Everybody says that time is this like most important resource. I argue that attention is the most important resource. I could be doing something specific with my time, but if my attention is elsewhere or a hundred percent of my attention is focused on Somethin…”

“And there's a very big difference between economics based on attention, right. Versus economics that are based on like trust and identity. And I actually think real creators build their businesses around trust and identity, not around attention.”

“People have found that the optimal ratio tends to be mostly SSM layers with a little bit of attention. So maybe a ratio of like 10 to one, I know of at least Probably like five groups that have independently verified that this is kind of the optimal ratio of t…”

“Most humans don't actually have much knowledge to share. They're just bullshitting, right, for attention.”

“And I would argue that attention is the most mispriced asset right now that most impacts everyone.”

“We ended up focusing a lot more on Memory reading and writing, because that turned out to be the majority of time when you're doing attention is reading and writing memory.”

“The constraining factor on humanity isn't how much physical capital there is. We can do amazing things. I mean, look at Germany. Germany just built a whole bunch of LNG terminals in, like, a few months, right? These are projects that people were like, oh, this…”

“Some of the most important things that we should be allocating attention to don't have prices associated with them, and cannot have prices associated with them, and so as a result, we cannot be relying on the market mechanism to guide our attention. We have to…”

“Flow states are simultaneously The deepest form of attention that a human being can provide. And once you get into it, they're the least difficult form of attention.”

“Digital technology is exactly like those two prior changes. It shifts the binding constraint and the binding constraint is no longer physical capital, but it is attention. Like what are humans paying attention to?”

“Technology is designed to get us to respond, and the way to do that is to heighten emotions and hijack attention, and it does so with ferocious efficiency, and it's unrelenting.”

“There are only two places your attention should be, either on the task at hand or on others. So if you're feeling any negative emotion you feel, whether it's doubt or fear or frustration or loneliness, it's a sign to redirect your attention.”