Self Attention
topic on 2 shows · 3 statements across 2 episodes
3 statements about Self Attention, every show
Yi Tay: The Architecture That Achieves AGI Will Still Be a Transformer
“It will be a transformer, I think. Like people, it depends on what you call it, but I think unless the paradigm shifts completely, which is, I mean, as a scientist, you cannot like completely say no to like that, this would never happen. But my feeling is that…”
Yi Tay: Efficient attention research consistently failed to eliminate self-attention
“There was this whole big era, which I was also involved in this era, where people try to like undermine the attention as much as possible. Like they try to remove it, simplify it, make it efficient, like this whole Like, efficient attention era. At the end of …”
Self-Attention Brought GPU Parallelism to Sequence Modeling
“The insight here was, Hey, you can use the same attention thing to like, look back at the past of the sequence that you're trying to produce. And you know, the beauty is that the, that It runs great on on GPUs and CPUs, and it's kind of parallel to, like, how …”