Ma: Scaling Web Data Unlikely to Solve Complex Math Conjectures
Tengyu Ma · No Priors Ep. 67 | With Voyage AI Co-Founder and CEO · Jun 6, 2024 · at 35:16
Tengyu Ma, Stanford professor and Voyage AI CEO, discusses why academic AI labs should focus on algorithmic innovations in reasoning rather than standard web scaling.
0:00 / 0:24exact quote · 24.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It's very unclear whether you can really the scaling law is really enough to get Get you to prove Riemann hypothesis or any of the math conjectures. So you know, and also you have to be superhuman performance in some sense, right? So if you turn on just the common crowd data on the web, can you be a good mathematician? It's kind of very hard to believe that. So, so we need more innovations there.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Tengyu Ma
Ma: RAG Will Remain Much Cheaper Than Long-Context Windows
“My prediction is that reg will be much cheaper than long contacts going forward.”
Ma: AI Is Running Out of Compute and Training Data
“My vision is that in the future the efficiency is very important because we are running out of data and compute. So we have to either use the data much better and use the compute much better.”
Ma: Sophia Optimizer Improves LLM Pre-Training Efficiency by 2x
“One of the paper we wrote last year was Sophia which we found, where we found that we have a nutrient optimizer, which can improve the training efficiency by two X for pre-training.”
Ma: Meta Achieved 1.6x Training Efficiency Gain Using Sophia Optimizer
“And recently I think one of the Facebook friends actually used this in their large scale multimodal training. And they found that in on that scale, I don't know exactly how many parameters there are, but I think, I assume it's kind of more than a hundred billi…”