Assertion Supported AI assessment confidence: 95% certainty 4/5 debate potential 2/5

Cubuk: Out-of-domain AI scaling power-law slopes can be practically useless

Doge Cubuk · Building an AI Physicist: ChatGPT Co-Creator’s Next Venture · Sep 30, 2025 · at 20:29

Ekin Dogus Cubuk is co-founder of Periodic Labs and former head of materials science at Google DeepMind. He discusses published research on out-of-domain generalization and scaling laws.

0:00 / 0:18exact quote · 18.9s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“We published a paper where we saw that as you increase the size of your training set, the IID performance, the in-domain performance improves the power law. Out-of-domain performance also improves the power law, but depending on what the outdoor domain is, like how far you are from training distribution, That power law might have such a small slope that is basically useless.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

Ekin Dogus Cubuk's research showed that while out-of-domain AI model performance scales as a power law with training set size, the power-law slope flattens significantly for tasks distant from the training distribution, making standard pre-training scaling practically useless for far-out-of-domain problems.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

Published research co-authored by Ekin Dogus Cubuk prior to September 2025 (such as the 2023 Nature paper 'Scaling deep learning for materials discovery' and 2024 TMLR paper 'Scaling (Down) CLIP') confirms that while out-of-domain performance scales as a power law with training dataset size, the slope/exponent flattens significantly as the evaluation target moves further from the pre-training data distribution.

2023 Nature paper co-authored by Ekin Dogus Cubuk ('Scaling deep learning for materials discovery') reported lower power-law exponents for distant out-of-distribution data regimes.
2024 TMLR paper ('Scaling (Down) CLIP') showed that distant OOD distributions exhibit significantly flatter scaling exponents compared to in-domain metrics.

Checked against

[1] cognitiverevolution.ai [2] cognitiverevolution.ai [3] cognitiverevolution.ai [4] repec.org [5] github.io [6] researchgate.net [7] researchgate.net [8] researchgate.net

"Ekin Dogus Cubuk" "power law" "materials discovery" OR "GNoME" "Ekin Dogus Cubuk" "power law" OR "scaling law" "out-of-domain" OR "out of domain" OR "out-of-distribution" "Ekin Dogus Cubuk" OR "Ekin Cubuk" "pre-training" "power law" OR "scaling" "Ekin Dogus Cubuk" "slope" "power law" OR "scaling" "Ekin Dogus Cubuk" pre-training transfer scaling power law Ekin Dogus Cubuk scaling laws out of domain power law slope "Ekin Dogus Cubuk" OR "Ekin Cubuk" "scaling" "out-of-distribution" OR "transfer"

assessed as of 2026-07-28

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

Doge Cubuk out-of-domain AI scaling

⚑ Spot an error? Report it

More from Doge Cubuk

Assertion Not checkable as stated

Cubuk: Frontier AI labs haven't trained LLMs on physics or chemistry

“The frontier AI labs have figured out how to train them on math and logic, but not yet on physics chemistry.”

Doge Cubuk Sep 30, 2025 ▶ 29:53 Building an AI Physicist: ChatGPT Co-Creator’s Next Venture

Opinion

Cubuk: A 200K superconductor would reshape fundamental quantum physics

“For example, if we could find a 200 Kelvin superconductor, even before we make any product with it, to be able to see such quantum effects at such high temperatures, I think would be such an update to people's view of how they see the universe.”

Doge Cubuk Sep 30, 2025 ▶ 0:33 Building an AI Physicist: ChatGPT Co-Creator’s Next Venture

Prediction Not checkable as stated

Cubuk: Quantum mechanics foundation models are AI's next frontier

“And we feel like teaching these LLMs to be foundation models, but for quantum mechanics will be the next frontier for LLMs.”

Doge Cubuk Sep 30, 2025 ▶ 10:05 Building an AI Physicist: ChatGPT Co-Creator’s Next Venture

Disclosure

Cubuk: Periodic Labs is building an automated powder synthesis lab

“We're going to have a powder synthesis lab, and turns out this is one of those methods where robots can do it, like very cheap, simple methods.”

Doge Cubuk Sep 30, 2025 ▶ 9:36 Building an AI Physicist: ChatGPT Co-Creator’s Next Venture

Insight

Cubuk: Physical experiments provide unhackable training signals for LLMs

“As we measure it, the LLM will get very clear signal. It's hard to hack, you know, unless, unlike these other LLM training techniques, it's like really what you see in real life is the signal that's going to the LLM.”

Doge Cubuk Sep 30, 2025 ▶ 13:10 Building an AI Physicist: ChatGPT Co-Creator’s Next Venture

Assertion Not checkable as stated

Cubuk: Existing superconductivity datasets are too noisy for AI training

“For superconductivity, there is a lot of data sets you can look at, but the noise floor on them is so high that training on them usually doesn't help.”

Doge Cubuk Sep 30, 2025 ▶ 22:00 Building an AI Physicist: ChatGPT Co-Creator’s Next Venture

Cubuk: Out-of-domain AI scaling power-law slopes can be practically useless

More from Doge Cubuk

Turn any episode into a week of clips.