Newton: Google's AI models are trained to 'teach to the test'
Casey Newton · Bold Predictions For Tech and AI in 2025 — With Casey Newton · Jan 5, 2025 · at 25:57
Casey Newton gives his theory on why Google's Gemini models lead public LLM benchmark leaderboards despite underwhelming product execution.
0:00 / 0:27exact quote · 27.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Well, I have a conspiracy theory, which is you know kind of not great to be sharing on a podcast, but I'm gonna do it anyway, which is, I think Google is teaching to the test. You know what I mean? Yes. Like, I think that they know what the benchmarks are, and they train the models to be really good at the benchmarks, and then they train the models, and they come out and they say, look, it does really well on the benchmarks, and then you just try to use it for other stuff, And it kind of sucks.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Casey Newton
Newton predicts Grammarly will be dead by the end of 2025
“Oh, can I give you a prediction? Grammarly dead by end of 2025.”
Newton: AI lab executives are setting expectations for linear progress in 2025
“So, based on the conversations that I've had with executives of the AI labs, it feels like people are trying to set expectations for more linear progress next year.”
Newton: Microsoft and OpenAI will stay partnered in 2025 despite tensions
“I mean, my, like, possibly contrarian take is, like, they just kind of muddle through. Like, that's kind of what I think that they're going to do. OpenAI needs Microsoft for the compute, the data centers, the, like, infrastructure of making AI. They're not goi…”
Newton predicts at least one AI company will reap massive LLM profits
“So when I look at AI, I see something that yes, is very expensive to train, but when you look at the fact that it's a general purpose technology and that we are finding more and more things to do with it, I do believe that at least one of these companies is go…”
Newton: Apple Intelligence is the flop of the year
“What I would say is Apple intelligence flop of the year.”
Newton: Apple Intelligence is built for users two years behind cutting-edge AI
“If you're, like, a tech nerd, like we are, and you want to live in the future, and you want to, like, use tools to improve your life, you would not be using any of the Apple intelligence stuff, because it, like, it truly is aimed at somebody who is two years b…”