Jun 20, 2023 · 1h 23m · latent-space
Ep 18: Petaflops to the People — with George Hotz of tinycorp
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this interview on the Latent Space podcast, George Hotz details the founding theses of tinycorp, the minimalist architecture of the Tinygrad framework, the engineering behind the Tinybox hardware, and his vision for democratized AI and digital immortality.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 17.4% of the talking time here. How this is scored →
The hosts as informed peer 5.1 Guest teaching 3.6 Guest disagreement 5.3 The hosts pushing back 3.4
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:19:00 Hotz dismisses e/acc and political discourse
Hotz bluntly dismisses e/acc as an unserious ideology and mocks tech figures who complain about basic political realities on social media.
Hardest push from the hosts ▶ 1:08:48 Swyx challenges video-based brain uploadingSwyx directly rejects Hotz's assertion that his YouTube streams constitute an uploaded brain, pointing out that public voice represents only a tiny fraction of mental state.
Biggest teaching moment ▶ 4:41 The non-NVIDIA training chip quizHotz challenges the hosts to name any non-NVIDIA chip successfully used for training, repeatedly shooting down their rapid-fire incorrect guesses until revealing TPUs.
The host holds their own ▶ 1:02:09 Swyx theorizes the Kanban lineSwyx demonstrates domain expertise by synthesizing Venkatesh Rao's API line concept with his own published theory about the Kanban line governing engineering management.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Core Theses: RISC ML, Hardware Co-Design, and Avoiding Turing Completeness | 3 | 7 | 6 | 2 | Hotz quizzes the hosts on which non-NVIDIA chips are actually used for ML training, shooting down their guesses of AMD, Intel, Cerebras, and Apple before leading them to TPUs. He then lectures deeply on why Turing completeness and systolic arrays are problematic in ML hardware. | |
| Tinygrad Architecture, Laziness, Operation Fusing, and Qualcomm Success | 5 | 3 | 4 | 2 | Hotz details the architectural choices of Tinygrad including laziness and operation fusion. The hosts keep pace by bringing up PyTorch Dynamo, Triton, and John Carmack's debugging philosophies. | |
| Overcoming AMD Driver Woes and Open-Source Software Culture | 5 | 2 | 5 | 3 | Hotz recounts his frustrations with AMD driver kernel panics and reaching out to Lisa Su. Swyx contrasts Hotz's break from PyTorch APIs with Chris Lattner's superset approach in Mojo. | |
| Engineering the Tinybox: Thermals, Power Limits, and Quantization | 6 | 4 | 5 | 4 | Hotz discusses hardware engineering trade-offs in Tinybox and rails against int4 quantization without proper loss metrics. Swyx pushes back on weight conversion lossless assumptions, prompting Hotz to concede uncertainty. | |
| Decentralized AI Hubs, Flopcoin, and Smarter Model Architectures | 6 | 3 | 6 | 3 | Hotz dismisses mixture of experts as an admission of running out of ideas and discusses PCIe bandwidth constraints. Swyx brings up parameter-efficient fine-tuning and the historical evolution of positional embeddings. | |
| The Bitter Lesson, Modern Hiring, and Embodied Robotics | 6 | 3 | 5 | 4 | Hotz invokes Rich Sutton's Bitter Lesson and explains his bounty-based hiring model. Swyx builds on the 'API line' framework by introducing his own concept of the Kanban board line above programmers. | |
| Vision for 2035: AI Companions, Digital Immortality, and Information Theory | 5 | 4 | 5 | 5 | Hotz explains his ambition to build an AI girlfriend and achieve digital immortality via YouTube compression. Swyx challenges him directly on whether recorded speech is sufficient representation of inner brain state. | |
| Philosophy, Transformers, Avatar 2 Script Rewrite, and Future Ambitions | 5 | 3 | 6 | 4 | Hotz critiques e/acc ideology and rewrites Avatar 2's script while discussing transformers as weight generation. Swyx presses him on why political accelerationism lacks seriousness compared to effective altruism. |
Statements from this episode (32)
Hotz: tinycorp will eventually partner or manufacture its own chips
“So I'd like to start another organization that eventually in the limit either works with people to make chips or makes chips itself and makes them available to anybody.”
Hotz: Tinygrad runs all ML models with only 25 primitive operations
“Tiny grad is, we are going to make a risk offset for all ML models. And yeah, it can run all ML models with basically 25 instead of the two 50 of XLA or PrimTorch. So about 10 X less complex.”
Hotz: Google TPUs are the only successful non-Nvidia training chips
“The only company, there's one other company aside from Nvidia who's succeeded at all at making training chips... Mid journey is trained on TPU, right? Like a lot of startups do actually train on TPUs, and they're the only other successful training chip aside f…”
Hotz: Analog computing for AI won't work and clockless chips aren't practical
“Analog computing just won't work. And clockless computing sure, it might work in theory, but your ETA tools are, maybe AIs will be able to design clockless chips, but not humans.”
Hotz: Systolic arrays are the wrong architectural choice for AI chips
“I think systolic arrays are the wrong choice. Systolic array, I think they have systolic arrays because that was the guy's PhD. And of course Amazon makes... They are very power efficient, but it becomes hard to schedule a lot of stuff. On them, if you're not …”
Hotz: Google's TPU compiler is a closed-source 32MB binary blob
“Not only is the chip closed source, But all of XLA is open source, but the XLA to TPU compiler is a 32 megabyte binary blob called lib TPU on Google's cloud instances. It's all closed source.”
Hotz: Tinygrad beats CoreML on ONNX tests and will soon pass ONNX Runtime
“We're below Onyx runtime, but we're beyond CoreML. So, like, that's, like, where we are in Onyx support now, but we will pass Onyx runtime soon, because it becomes very easy to add ops, because of how, like, you don't need to do anything at the lower levels.”
Hotz: Tinygrad is about 5x slower than PyTorch on Nvidia GPUs
“The correctness for both forwards and backwards passes is there, but on Nvidia, it's about five X slower than PyTorch right now.”
Hotz: Tinygrad runs OpenPilot in production 2x faster than Qualcomm's library
“TinyGrad is used to run the model in OpenPilot. Like right now, it's been live in production now for six months. And TinyGrad is about two X faster on the GPU than Qualcomm's library.”
Hotz: Intel GPUs feature stable kernel drivers and public register docs
“Intel GPUs have a stable kernel driver and they have all their hardware documented. You can go and you can find all the register docs on Intel GPUs.”
Hotz: AMD CEO Lisa Su sent pre-release ROCm 5.6 to fix panics
“Lisa Sue reached out, connected with a whole bunch of different people. They sent me a pre-release version of Rock M 5.6. They told me you can't release it, which I'm like, okay, Why do you care? But they say they're going to release it by the end of the month…”
Hotz: Consumer AMD GPUs lack peer-to-peer support
“If you have a consumer AMD GPU, they don't support peer to peer.”
Hotz: Tinygrad could replicate PyTorch's API in two engineer-months
“Replicating the PyTorch API. Is something I can do with a couple, you know, like an engineer month or two.”
Hotz: Nobody has successfully trained models in INT8
“No one's gotten training to work with Indate yet. There's a few papers that vaguely show it, but if you're training, you're going to need BF-sixteen or float-sixteen.”
Hotz: None of 20 tested PCIe extenders work at PCIe 4.0
“No PCI extender I've tested and I've bought 20 of them works at PCIe four point out. So you're going to need PCIe redrivers now.”
Hotz: Halving GPU power yields 80% of peak performance
“Now, you can limit power on GPUs and still get, you can use like half the power and get 80% of the performance. This is a known fact about GPUs”
Hotz: Best chatbots will be smaller models with 1,000 training runs
“I don't think that the best chatbot models are going to be the big ones. I think the best chatbot models are going to be the ones where you had a thousand training runs instead of one. And I don't think that the interconnect bandwidth is going to matter that m…”
Hotz: Tinybox will be 5x faster than H100 systems per dollar
“For 90% of most companies model training use cases, the tiny box will be five X faster for the same price.”
Hotz: GPT-4 is an 8-way mixture model with 220B parameters per head
“GPT-IV is two hundred twenty billion in each head, and then it's an eight-way mixture model.”
Hotz: Except for Apple, secretive tech companies hide underwhelming tech
“Whenever a company is secretive, with the exception of Apple, Apple's the only exception, whenever a company is secretive, it's because they're hiding something that's not that cool.”
Hotz: Meta attracts researchers who want to publish while OpenAI keeps ideologues
“OpenAI can keep ideologues who, you know, believe ideological stuff, and Facebook can keep every researcher who's like, dude, I just want to build AI and publish it.”
Hotz: Machines will replace all human labor in about 20 years
“I'm a believer that machines are gonna replace everything in about 20 years.”
Hotz: Training on internet cross-entropy loss yields mediocre AI responses
“The problem is, if your loss function is categorical across entropy on the internet, your responses will always be mid.”
Hotz: RLHF models adopt customer support personalities
“I don't like the RLHF models. I don't like the tuned versions of them. I think that they become, you take on the personality of a customer support agent, right?”
Hotz: Third company will build an AI girlfriend product
“The third company's the first one that's gonna build a real product, and that product is A girlfriend? No, like I'm dead serious, right? Like this is the dream product, right? This is the absolute dream product.”
Hotz: Merging with machines requires an AI companion, not Neuralink electrodes
“So I don't need to put, you know, electrodes in my brain to merge with a machine. I need an AI girlfriend, right?”
Hotz: Maximally compressed human brain represents only a couple gigabytes
“Quantization is a poor man's compression. I think we're only talking really here about, like, maybe a couple gigabytes, right? And then if you have, like, a couple gigabytes of true information of yourself up there, cool man. Like, what does it mean for me to …”
Hotz: Real AI alignment problem is corporate and government misalignment
“I think it's actually not a question of whether the computer is aligned with the company who owns the computer. It's a question of whether that company's aligned with you or that government's aligned with you. And the answer is no. And that's how you end up de…”
Hotz: Qualcomm SNPE cannot run transformers due to missing outer product ops
“Qualcomm's S and PE can't run transformers for this reason. So most matrix multiplies in neural networks are weights times values, right? Whereas you know, when you get to the outer product in in transformers, well, it's waste times weight. It's a, it's values…”