Karpathy: llm.c is only about 3,000 lines of code
Andrej Karpathy · llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE · Sep 21, 2024 · at 21:11
This episode carries Andrej Karpathy's own address, with nobody on the show putting questions to them. It still counts as said, and it is kept out of every score on their page.
Andrej Karpathy highlights the simplicity and readability of his pure C/CUDA training library, llm.c.
0:00 / 0:03exact quote · 4.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It's only maybe like, I think, 3000 lines of code, basically C mostly.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Andrej Karpathy
Karpathy predicts LLMs will act as compilers generating bare-metal CUDA code
“If LLINs are about to become much better at coding over time, then I think you can expect that the LLIN could actually do this for any custom application over time. And so the LLINs could act as a kind of compiler What you're interested in, they're gonna do al…”
Karpathy: Python and PyTorch are crutches for finite human intelligence
“The use of Python and PyTorch and everything else is just a crutch, because we humans are finite. We have finite knowledge, intelligence, and attention.”
Karpathy: llm.c trains GPT-2 on one H100 node in 24 hours for $600
“You can train it on a single node of H-one-hundreds in about 24 hours, and that costs roughly 600 dollars.”
Karpathy: llm.c was 20% faster and used 30% less memory than PyTorch
“At the time of that post, we were using, in LL and that's in 30% less memory, and we were 20% faster in training, just the truth.”
Karpathy: llm.c avoids tensor abstractions in favor of raw float arrays
“And one thing I'd really like to do in LN.c is I just want to keep things simple. I don't want to create a tensor abstraction. I don't want to create any abstraction, really. It's just float arrays and operations on float arrays.”
Karpathy: The popular PMPP textbook lacks advanced CUDA optimization techniques
“PMPP is actually quite good but also, I think, still kind of like mostly on the beginner level, because a lot of the CUDA code that we ended up developing in the lifetime of the LMC project, you would not find those things in, in this book, actually.”