The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 14 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Karpathy predicts LLMs will act as compilers generating bare-metal CUDA code
“If LLINs are about to become much better at coding over time, then I think you can expect that the LLIN could actually do this for any custom application over time. And so the LLINs could act as a kind of compiler What you're interested in, they're gonna do al…”
Karpathy: Python and PyTorch are crutches for finite human intelligence
“The use of Python and PyTorch and everything else is just a crutch, because we humans are finite. We have finite knowledge, intelligence, and attention.”
Karpathy: llm.c trains GPT-2 on one H100 node in 24 hours for $600
“You can train it on a single node of H-one-hundreds in about 24 hours, and that costs roughly 600 dollars.”
Karpathy: llm.c was 20% faster and used 30% less memory than PyTorch
“At the time of that post, we were using, in LL and that's in 30% less memory, and we were 20% faster in training, just the truth.”
Karpathy: llm.c avoids tensor abstractions in favor of raw float arrays
“And one thing I'd really like to do in LN.c is I just want to keep things simple. I don't want to create a tensor abstraction. I don't want to create any abstraction, really. It's just float arrays and operations on float arrays.”
Karpathy: The popular PMPP textbook lacks advanced CUDA optimization techniques
“PMPP is actually quite good but also, I think, still kind of like mostly on the beginner level, because a lot of the CUDA code that we ended up developing in the lifetime of the LMC project, you would not find those things in, in this book, actually.”
Karpathy: llm.c rejects performance PRs if added complexity harms developer onboarding
“A lot of lmc just kind of, like balancing the improvement and speed with the complexity of what you're actually introducing, and so I've actually rejected a lot of PRs because of that, because the code starts to get crazy, and I think that decreased the amount…”
Karpathy: llm.c trains transformers in C with minimal C++
“We're training transformers in C at a pinch of C++.”
Karpathy: Without PyTorch, developers are 'naked in the world'
“So, PyTorch is really, really nice, and this is just some of the things that PyTorch offers. So, without PyTorch, we're kind of naked in the world, right?”
Karpathy: llm.c avoids runtime crashes by pre-allocating memory statically without dependencies
“It's a single file C. There's no dependencies whatsoever. It compiles instantly. It runs instantly. All the memory is just allocated in a single blob. So if you start stepping, there's no way you're gonna boom later. It's all preplanned. It's fully determinist…”
Karpathy: llm.c was created while jet-lagged on vacation in the Maldives
“All of this work that I described so far, I was on vacation and while I was jet-lagged in Maldives. So I, basically it's perfect because you wake up at one a.m., and there's nothing to do. So you write stuff like LLN.C.”
Karpathy: llm.c achieves nearly 50% MFU on single-node GPT-2 training
“We have almost a 50% NFU here on one node, which is quite good.”