Hotz: GPT-4 is an 8-way mixture model with 220B parameters per head
George Hotz · Ep 18: Petaflops to the People — with George Hotz of tinycorp · Jun 20, 2023 · at 49:48
George Hotz details OpenAI's unconfirmed architecture details for GPT-4 during a discussion on foundation model design.
0:00 / 0:04exact quote · 4.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“GPT-IV is two hundred twenty billion in each head, and then it's an eight-way mixture model.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from George Hotz
Hotz: Meta attracts researchers who want to publish while OpenAI keeps ideologues
“OpenAI can keep ideologues who, you know, believe ideological stuff, and Facebook can keep every researcher who's like, dude, I just want to build AI and publish it.”
Hotz: Analog computing for AI won't work and clockless chips aren't practical
“Analog computing just won't work. And clockless computing sure, it might work in theory, but your ETA tools are, maybe AIs will be able to design clockless chips, but not humans.”
Hotz: Systolic arrays are the wrong architectural choice for AI chips
“I think systolic arrays are the wrong choice. Systolic array, I think they have systolic arrays because that was the guy's PhD. And of course Amazon makes... They are very power efficient, but it becomes hard to schedule a lot of stuff. On them, if you're not …”
Hotz: Tinybox will be 5x faster than H100 systems per dollar
“For 90% of most companies model training use cases, the tiny box will be five X faster for the same price.”
Hotz: Machines will replace all human labor in about 20 years
“I'm a believer that machines are gonna replace everything in about 20 years.”
Hotz: Maximally compressed human brain represents only a couple gigabytes
“Quantization is a poor man's compression. I think we're only talking really here about, like, maybe a couple gigabytes, right? And then if you have, like, a couple gigabytes of true information of yourself up there, cool man. Like, what does it mean for me to …”