Amodei: Python made up only 0.1% to 1% of GPT-3's training data
Dario Amodei · Improving AI with Anthropic's Dario Amodei · Sep 25, 2023 · at 5:21
Dario Amodei, CEO of Anthropic and former lead researcher at OpenAI, describes how surprisingly little Python data was required in GPT-3's web scrape for emergent coding capabilities to appear.
0:00 / 0:09exact quote · 9.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“When we looked through it was like, you know, it's hard to estimate, but it was something like, you know, maybe .1% to one percent of the data that we scraped was Python data.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dario Amodei
Amodei: AI training costs will hit $1B in 2024 and up to $10B in 2025
“I think that next year we're probably going to see from multiple players models on the order of one billion dollars, and in 2025 we're going to see models on the order of Several billion, I don't know, perhaps even ten billion dollars.”
Amodei: Bureaucratic AI regulation will cause the West to lose to adversaries
“If this is like, you have to fill out a thousand pages of paperwork and get 15 different licenses from different bodies to, you know, to make an AI system, that's never gonna work, that's gonna slow things down, you know, other adversaries You know, authoritar…”
Amodei: AI scaling laws will continue driving improvements without algorithmic breakthroughs
“I think we are on track, even if there were no algorithmic improvements from here, even if we just scaled up what we had so far, ah, I think the scaling laws are going to continue, and I think that's going to lead to amazing improvements.”
Amodei: Next-word prediction scaling has no limits and will continue
“Our view was that, look, this is the beginning of something amazing because there's no limit and you can continue to scale it up. And there's no reason why the patterns we've seen before won't continue to hold. The objective of predicting the next word is so r…”
Amodei: AI inference will remain affordable over next 3-4 years
“I think my basic view is that inference will not get that much more expensive. The simple, you know, the basic logic of the scaling laws is that if you increase compute by a factor of n, you need to increase data by a factor of square root of n. And size the m…”
Amodei: Generalists outperform long-time domain experts in fast-moving AI
“AI was and still is to some extent very young and is definitely moving very fast. And so when that's the case, really talented generalists can often outperform those who have been in the field for a long time because things are being shaken up so much. If anyt…”