Hardware and software co-evolve to favor dominant AI architectures
Tri Dao · FlashAttention-2: Making Transformers 800% faster AND exact · Aug 3, 2023 · at 36:22
FlashAttention creator Tri Dao discusses how AI hardware and software frameworks co-evolve to reinforce dominant architectures like Transformers.
0:00 / 0:22exact quote · 22.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“There is this feedback loop where somehow The model architectures that take advantage of hardware become popular, and the hardware will also kind of evolve to optimize a little bit for that kind of architecture, and software framework software frameworks will also evolve to, like, optimize for that particular architecture.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Tri Dao
Compilers will automate complex kernel fusion within two years
“Maybe in a year or two, we'll, we'll have compilers that are able to do a lot of these optimizations for you, and you don't have to, for example, spend a couple months writing CUDA to get this stuff to work.”
RNNs will outperform Transformers in batch generation and long sequences
“I am personally bullish on, on, on RNNs. I think RNNs they don't, they essentially summarize the past into a state vector. They have fixed size, so the size doesn't grow with the history. So that means that you don't need as much memory to keep around all the …”
LLaMA 2 will shift developers from closed APIs to self-hosting
“And I do see that's going to shift the balance of it. More and more folks are going to be using let's say derivatives of Lama two. More folks are going to Fine-tune and serve their own model instead of calling an API.”
FLOP counts do not necessarily correlate with wall-clock runtime
“Flops or floating point operations don't necessarily correlate with runtime. There are other factors like memory reading and writing, parallelism, and so on.”