GPT III
product on 9 shows · 16 statements across 16 episodes
the Y Combinator Startup Podcast 
More or Less 
American Optimist 
Latent Space 
No Priors 
Sourcery 
How I Built This 
the a16z Podcast 
Big Technology
16 statements about GPT III, every show
Brown: GPT-3 Capabilities Could Not Scale With Test-Time Compute Budget
“Like, with GPT-III, you couldn't scale test time compute. Like, if you gave it a budget of ten million dollars and said, okay, well, let's see what GPT-III can do, it really can't do that much, more than what you could do with, like, 10 dollars or one dollar.”
Andreessen: AI Dungeon was the only public way to access GPT-3 for a year
“There was like a year where like the only way for a normal person to use GPT-III was in an AI dungeon.”
Shadbolt: The public never cares about supercomputers, only their output
“People have been building leadership class supercomputers for a very long time, and usually people don't care about them, and that's fine. Like nobody cared prior to GPT-III. Nobody cared that Azure and Microsoft and open AI were building big, ugly computers. …”
Mohan: Training optimizer state requires 14x model parameter size in memory
“The optimizer state. So while you're training is 14 times the size of the model. So in this case, if it's like one hundred seventy billion parameters, it's probably, I'm not great at mental math here, but that's probably around 2.5 terabytes to just store the …”
Alidoust: Protein AI is past GPT-3 while cell models are at GPT-1
“I think in the protein, protein models, we are past GPT-III.
when it comes to single cell models and virtual cell models, yeah, I think GPT-I to two right now.
I think we're closer to GPT-I than two.”
Hu: 10T parameter models will spark a GPT-3 level innovation leap
“I think the type of level of potential innovation could be the same leap we saw from GPT-II, which was around one billion Parameters that was released with the paper of a scaling loss, which was one of these seminal papers that people figure out, okay, this is…”
Davis: OpenAI trained GPT-3 on 10,000 V100s for 14.6 days
“GPT-III was trained on 10,000 V-one-hundred GPUs in an interconnected cluster in Azure for about 14.6 days.”
Gil: Nobody wanted to start an OpenAI competitor after GPT-3
“And I remember when GPT-III came out, I tried to both pitch a bunch of people to start effectively an open AI competitor, and nobody wanted to do it, because they didn't think it was a good business model, as well as it was such a big technology step up if you…”
Quora found GPT-3 could not match top human answer quality
“We ran some experiments using GPT-III to generate answers and compare them to the answers that, that humans had written on Quora. And a lot of the time, GPT-III could not write as good of an answer as what the best human answer was that, that, that had been wr…”
Park: GPT-3 simulated COVID-19 discussions without prior pandemic training data
“We basically asked GPT-III to create a community that has to talk about COVID and vaccination, vaccination policy. And you would wonder, it shouldn't be able to do that in theory, because it doesn't know anything about COVID. It doesn't know anything about the…”
Masad: Machine learning on code did not work effectively until GPT-2/GPT-3
“Every year or two while starting Replit, I would go look at the state of the art on ML on code and nothing ever really worked all that well up until GPT-II. And you can like take GPT-II and fine tune it and like try to make it write code and was like kind of o…”
McGrew: GPT-3 and GPT-4 must be prompted to act smart or dumb
“Well, that's actually one of the cool things that we've discovered with GPT-III and GPT-IV, Is that the model, so the models are trained on the internet and they're reading everything, right? Then they try to act like what they've seen, right? They're predicti…”
Habib: OpenAI's 1.3B InstructGPT beat the 100x larger GPT-3 via RLHF
“In the InstructGPT paper that OpenAI released, they compared, you know, a one or two billion parameter model with instruction tuning and RHF to the full GPT-III model and people preferred that despite the fact it was a hundred times smaller.”
Chris Power: AI compute depends directly on Taiwanese semiconductor security
“You know, even, even if you're an EA type of person or, you know, you're a pacifist or whatever, or if you're into climate change or like, you know, pick, pick any industry that you think is important. Like good luck running compute on GPT-III if there are no …”
Altman: Models orders of magnitude above GPT-3 could enable massive disinformation
“Like, at some point there will come a time, one, two, three orders magnitude, who knows what, how much more powerful than GPT-III, where I think a language model Can really have a huge disinformation effect on the world.”
Marcus: No established methodology exists for debugging chatbots or autonomous vehicles
“None of these chatbot systems are well-debugged. Nobody knows how to debug them, in fact. And so both the problem with GPT-III and with driverless cars is we don't actually have a methodology even for debugging it.”