Kernel fusion sacrifices flexibility for researchers experimenting with attention
Tri Dao · FlashAttention-2: Making Transformers 800% faster AND exact · Aug 3, 2023 · at 10:09
Tri Dao, creator of FlashAttention, explains the trade-offs of kernel fusion in machine learning architectures during a discussion on GPU memory optimization.
0:00 / 0:32exact quote · 32.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“When you do kernel fusion is a little bit you lose a little bit of flexibility in the sense that, hey, now you have for example, is flash attention is just a subroutine that you would call to do attention. But as a researcher, let's say you don't want that exact thing, right? You don't want just attention. Let's say you want some modification to attention... And so kernel fusion just means that, okay, and we have a subroutine that does the entire thing, but if you want to experiment with things, you won't be able to use that that fused kernel.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Tri Dao
Compilers will automate complex kernel fusion within two years
“Maybe in a year or two, we'll, we'll have compilers that are able to do a lot of these optimizations for you, and you don't have to, for example, spend a couple months writing CUDA to get this stuff to work.”
RNNs will outperform Transformers in batch generation and long sequences
“I am personally bullish on, on, on RNNs. I think RNNs they don't, they essentially summarize the past into a state vector. They have fixed size, so the size doesn't grow with the history. So that means that you don't need as much memory to keep around all the …”
LLaMA 2 will shift developers from closed APIs to self-hosting
“And I do see that's going to shift the balance of it. More and more folks are going to be using let's say derivatives of Lama two. More folks are going to Fine-tune and serve their own model instead of calling an API.”
FLOP counts do not necessarily correlate with wall-clock runtime
“Flops or floating point operations don't necessarily correlate with runtime. There are other factors like memory reading and writing, parallelism, and so on.”