Sep 21, 2024 · 23m · latent-space
llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this CUDA MODE presentation, Andrej Karpathy chronicles the development of llm.c—a minimalist, dependency-free framework for training transformer models in pure C and CUDA—and explores how bare-metal implementations preview the future of AI-driven code generation and domain-specific compilers.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 0.0 Guest teaching 1.9 Guest disagreement 1.1 The hosts pushing back 0.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 3:35 Rejection of black-box framework errors
Karpathy articulates his frustration at opaque compiler failures, leading to his decision to abandon high-level frameworks and build from scratch.
Hardest push from the hosts ▶ 2:20 Challenging modern deep learning abstraction layersKarpathy pushes back against standard multi-layered ML abstractions, pointing out that developers often do not understand what compilers do to underlying models.
Biggest teaching moment ▶ 5:20 Demystifying LayerNorm implementation in raw CKarpathy educates the audience on stripping out complex tensor abstractions to implement exact forward and backward passes using flat float pointers.
The host holds their own ▶ 0:00 Host contextualizes Karpathy's legacyThe host demonstrates deep familiarity with Karpathy's career contributions across OpenAI, ImageNet, and Tesla Autopilot.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Origins of llm.c and PyTorch Compiler Frustrations | 0 | 1 | 2 | 0 | Andrej gives a monologue describing his initial frustration with cryptic PyTorch compiler errors during GPT training, which provoked him into writing the implementation in raw C. The host is silent during the presentation. | |
| Deconstructing PyTorch Abstractions and Setting Reference Baselines | 0 | 2 | 1 | 0 | Andrej breaks down the foundational abstractions provided by PyTorch and demonstrates how to translate LayerNorm forward and backward passes into bare float arrays in C without tensor wrappers. | |
| End-to-End CPU Implementation and the Maldives Origin | 0 | 2 | 0 | 0 | Andrej outlines the CPU implementation, single-allocation memory model, zero-dependency philosophy, and his jet-lagged hacking session in the Maldives. | |
| Initial GPU Acceleration and the Learning Curve of CUDA | 0 | 2 | 2 | 0 | Andrej reviews moving from CPU to GPU kernels and shares mild critiques regarding outdated CUDA literature and the opacity of official programming guides for modern ML workflows. | |
| Community Collaboration and Open-Source Momentum | 0 | 2 | 1 | 0 | Andrej highlights open-source contributions from community members and discusses the trade-off between optimization and code readability, including nuking streams to prevent race conditions. | |
| GPT-2 Reproduction Milestones and Benchmarking Against PyTorch | 0 | 2 | 1 | 0 | Andrej presents GPT-2 reproduction benchmarks on 8xH100 GPUs, noting a 30% reduction in memory usage and 20% speedup over baseline PyTorch. | |
| Project Roadmap, Community Forks, and LLaMA Architecture Support | 0 | 2 | 1 | 0 | Andrej discusses future directions like LLaMA 3 and FP8 support, proposing the thesis that human-level framework abstractions like PyTorch may eventually be bypassed by LLMs compiling directly to custom C/CUDA. |
Statements from this episode (14)
Karpathy: llm.c trains transformers in C with minimal C++
“We're training transformers in C at a pinch of C++.”
Karpathy: Without PyTorch, developers are 'naked in the world'
“So, PyTorch is really, really nice, and this is just some of the things that PyTorch offers. So, without PyTorch, we're kind of naked in the world, right?”
Karpathy: llm.c avoids tensor abstractions in favor of raw float arrays
“And one thing I'd really like to do in LN.c is I just want to keep things simple. I don't want to create a tensor abstraction. I don't want to create any abstraction, really. It's just float arrays and operations on float arrays.”
Karpathy: llm.c avoids runtime crashes by pre-allocating memory statically without dependencies
“It's a single file C. There's no dependencies whatsoever. It compiles instantly. It runs instantly. All the memory is just allocated in a single blob. So if you start stepping, there's no way you're gonna boom later. It's all preplanned. It's fully determinist…”
Karpathy: llm.c was created while jet-lagged on vacation in the Maldives
“All of this work that I described so far, I was on vacation and while I was jet-lagged in Maldives. So I, basically it's perfect because you wake up at one a.m., and there's nothing to do. So you write stuff like LLN.C.”
Karpathy: The popular PMPP textbook lacks advanced CUDA optimization techniques
“PMPP is actually quite good but also, I think, still kind of like mostly on the beginner level, because a lot of the CUDA code that we ended up developing in the lifetime of the LMC project, you would not find those things in, in this book, actually.”
Karpathy: llm.c rejects performance PRs if added complexity harms developer onboarding
“A lot of lmc just kind of, like balancing the improvement and speed with the complexity of what you're actually introducing, and so I've actually rejected a lot of PRs because of that, because the code starts to get crazy, and I think that decreased the amount…”
Karpathy: llm.c trains GPT-2 on one H100 node in 24 hours for $600
“You can train it on a single node of H-one-hundreds in about 24 hours, and that costs roughly 600 dollars.”
Karpathy: llm.c achieves nearly 50% MFU on single-node GPT-2 training
“We have almost a 50% NFU here on one node, which is quite good.”
Karpathy: llm.c was 20% faster and used 30% less memory than PyTorch
“At the time of that post, we were using, in LL and that's in 30% less memory, and we were 20% faster in training, just the truth.”
Karpathy: llm.c will support LLaMA 3.1 training very soon
“We will have Lama 3.1 training in Lama.c very, very soon.”
Karpathy: llm.c is only about 3,000 lines of code
“It's only maybe like, I think, 3000 lines of code, basically C mostly.”
Karpathy predicts LLMs will act as compilers generating bare-metal CUDA code
“If LLINs are about to become much better at coding over time, then I think you can expect that the LLIN could actually do this for any custom application over time. And so the LLINs could act as a kind of compiler What you're interested in, they're gonna do al…”