Assertion Supported AI assessment confidence: 95% certainty 4/5 debate potential 2/5

Morris: Language models hit a hard memorization plateau regardless of dataset scaling

Jack Morris · Information Theory for Language Models: Jack Morris · Jul 2, 2025 · at 38:32

Cornell Tech PhD researcher Jack Morris explains his empirical findings on measuring the exact data storage capacity limits of neural language models.

0:00 / 0:09exact quote · 9.5s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“Like, no matter how you scale the training size, you hit this like perfect, perfect ish plateau in auto memorization, which we call the model capacity.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

Regardless of how training data size is scaled, language models trained on random data hit a distinct plateau in memorization capacity.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

Published empirical research led by then-researcher John X. Morris and co-authors from Meta FAIR, Google DeepMind, Cornell, and NVIDIA ('How much do language models memorize?', arXiv:2505.24832, May 2025 / ICML 2026) directly evaluated GPT-style transformer models ranging from 100K to 1.5B parameters trained on synthetic uniform random data. The study demonstrated that total memorized bits plateau at a fixed empirical capacity ceiling of approximately 3.6 bits per parameter (in 16-bit precision) once training data exceeds the model's capacity threshold, regardless of continued scaling of training set size. Subsequent independent follow-up research (e.g., Liu, 2026) and ICML peer-reviewed findings confirmed this robust capacity plateau across architectures and data scaling regimes.

Morris et al. (May 2025; ICML 2026 Outstanding Paper Honorable Mention) evaluated hundreds of transformer models on synthetic uniform random bitstrings/tokens and found memorization saturated at ~3.6 bits per parameter.
The empirical evaluations showed that once the dataset size scaled past a model's parameter capacity, the total memorized bits plateaued at a fixed ceiling rather than continuing to increase.
Independent studies in early 2026 (e.g., Liu, 2026) corroborated that models trained on random data reach a sharp, distinct memorization capacity ceiling scaling strictly with parameter count.

Checked against

[1] jxmo.io [2] researchgate.net [3] venturebeat.com [4] openreview.net [5] aimodels.fyi [6] scribd.com [7] arxiv.org [8] arxiv.org

"empirical capacity" "3.6 bits" "random data" "plateau" "How much do language models memorize" "John X. Morris" "random data" capacity bits Jack Morris memorization capacity language models plateau random tokens "Jack Morris" "memorization" "random data" OR "random" language models capacity "memorization capacity" "bits per parameter" "random data" plateau OR saturation "How much do language models memorize" "Section 3" "capacity" "bits per parameter"

assessed as of 2026-09-02

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

Jack Morris language models model capacity

⚑ Spot an error? Report it

More from Jack Morris

What-if

Morris: ChatGPT could likely have been built using RNNs instead of Transformers

“And I think like, we honestly probably could have gotten this with RNNs. I know like the scaling laws paper shows that RNNs have worse curves for scaling, but probably people would have been like, I bet you could have built chat GPT with a very sophisticated R…”

Jack Morris Jul 2, 2025 ▶ 1:10:28 Information Theory for Language Models: Jack Morris

Assertion Not checkable as stated

Jack Morris: Fundamental AI science shifted to companies due to academic compute limits

“That's when I think things really started to change in terms of the types of questions you wanted to ask can't always be answered with academic resources. So a lot of the like fundamental kind of like boundary pushing and AI science moved into companies.”

Jack Morris Jul 2, 2025 ▶ 4:22 Information Theory for Language Models: Jack Morris

Insight

Morris: AI paradigm shifts are driven by novel datasets, not architectures

“I think, like, all of the things that I would consider paradigm shifts in the Kuhnian sense came from a new technique, but trained on new data, and I think the new data is super, super important”

Jack Morris Jul 2, 2025 ▶ 1:09:20 Information Theory for Language Models: Jack Morris

Assertion Not checkable as stated

Jack Morris: Most AI research was previously open, but is now closed

“Most stuff was open. Now most stuff is not open.”

Jack Morris Jul 2, 2025 ▶ 3:39 Information Theory for Language Models: Jack Morris

Insight

Morris: Re-implementing new paradigm shifts quickly is the best AI grad strategy

“Honestly, if I were to give advice to a younger grad student, I think the way to do it would be literally just like sit and wait until the next kind of paradigm shift and then just immediately start working as fast as you can to like re-implement it. Like, I d…”

Jack Morris Jul 2, 2025 ▶ 5:28 Information Theory for Language Models: Jack Morris

Opinion

Morris: Two years of academic AI research on small models was inconsequential

“There was like kind of two years where everyone in academia was working on like smaller models and none of it really mattered.”

Jack Morris Jul 2, 2025 ▶ 8:36 Information Theory for Language Models: Jack Morris

Morris: Language models hit a hard memorization plateau regardless of dataset scaling

More from Jack Morris

Turn any episode into a week of clips.