Hu: 10T parameter models will spark a GPT-3 level innovation leap
Diana Hu · The 10 Trillion Parameter AI Model With 300 IQ · Y Combinator · Nov 1, 2024 · at 2:27
Diana Hu, partner at Y Combinator, compares the potential jump to 10-trillion-parameter AI models to the historical scaling leap from GPT-2 to GPT-3.
0:00 / 1:06exact quote · 66.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think the type of level of potential innovation could be the same leap we saw from GPT-II, which was around one billion Parameters that was released with the paper of a scaling loss, which was one of these seminal papers that people figure out, okay, this is transformer architecture that we figured out. What if we just throw a bunch of engineering and just do a lot of it? Where does this scale and this logarithmic type of scaling? Then this was proofed out when GPT-III got released. That was about a hundred seventy-ish billion parameters. So that's like that to order magnitude. And we saw what happened with that. That created this new flourishing era of AI companies, and we saw it. We experienced this back in 23, when we started seeing all these companies building on top of GBT 3.5 that was starting to work, and it created this giant wealth. So we could probably expect if this scaling law continues, the feeling will be similar to what we felt from that year of transition of 22 to 23.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Diana Hu
Diana Hu: Supabase is the default backend recommendation across all LLMs
“Whenever someone asks how to set up anything that you need, some sort of backend, Firebase type of transaction, the default answer from all the LLMs is actually a Superbase.”
Hu: Anthropic surpasses OpenAI as top API for YC applicants
“And, shockingly, in this batch, the number one API is actually Anthropic. It came out a bit more than OpenAI, which who would have thought?”
Diana Hu: AI's future Google- and Facebook-scale startups have not started yet
“Startups like the future Facebook or the future Google are yet to be started because those come in in the deployment phase, because right now I think these things, things are still getting built up.”
Hu: A YC startup's 8B model beat OpenAI on healthcare benchmarks
“There's this particular YCE startup that told me that they collected the best data set for healthcare. And they ended up performing better than OpenAI and a lot of the benchmarks for healthcare with only eight billion parameters.”
Hu: Cursor scaled from $1M to $100M revenue in one year
“We've seen crazy growth unlike anything, only possible with, right now with AI, like all these companies that we work with, zero to twelve million in 12 months. The cursor story where they went zero to one in one year, the next one to a hundred. This is, like,…”