Karpathy: llm.c was created while jet-lagged on vacation in the Maldives
Andrej Karpathy · llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE · Sep 21, 2024 · at 10:03
This episode carries Andrej Karpathy's own address, with nobody on the show putting questions to them. It still counts as said, and it is kept out of every score on their page.
Andrej Karpathy recounts the personal origin story and setting where he authored the initial version of llm.c.
0:00 / 0:13exact quote · 13.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“All of this work that I described so far, I was on vacation and while I was jet-lagged in Maldives. So I, basically it's perfect because you wake up at one a.m., and there's nothing to do. So you write stuff like LLN.C.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Andrej Karpathy
Karpathy predicts LLMs will act as compilers generating bare-metal CUDA code
“If LLINs are about to become much better at coding over time, then I think you can expect that the LLIN could actually do this for any custom application over time. And so the LLINs could act as a kind of compiler What you're interested in, they're gonna do al…”
Karpathy: Python and PyTorch are crutches for finite human intelligence
“The use of Python and PyTorch and everything else is just a crutch, because we humans are finite. We have finite knowledge, intelligence, and attention.”
Karpathy: llm.c trains GPT-2 on one H100 node in 24 hours for $600
“You can train it on a single node of H-one-hundreds in about 24 hours, and that costs roughly 600 dollars.”
Karpathy: llm.c was 20% faster and used 30% less memory than PyTorch
“At the time of that post, we were using, in LL and that's in 30% less memory, and we were 20% faster in training, just the truth.”
Karpathy: llm.c avoids tensor abstractions in favor of raw float arrays
“And one thing I'd really like to do in LN.c is I just want to keep things simple. I don't want to create a tensor abstraction. I don't want to create any abstraction, really. It's just float arrays and operations on float arrays.”
Karpathy: The popular PMPP textbook lacks advanced CUDA optimization techniques
“PMPP is actually quite good but also, I think, still kind of like mostly on the beginner level, because a lot of the CUDA code that we ended up developing in the lifetime of the LMC project, you would not find those things in, in this book, actually.”