Assertion Supported AI assessment confidence: 85% certainty 3/5 debate potential 2/5

MoE models with 20B active parameters match 72B dense models in memorization

Pratyush Maini · ⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology · Feb 10, 2026 · at 5:24

Pratyush Maini explains how Mixture-of-Experts (MoE) architectures replicate the memorization capacity of much larger dense models.

0:00 / 0:24exact quote · 24.1s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“With the coming of the MOE models, we're starting to see these phenomena, even like models which have a less number of active parameters. So a phenomena that we would observe at a 72 B model in the past is probably now visible at a 20 B active one 20 B MOE also. So in some way the active parameters might be lesser in the MOE life cycle, but still be able to like regurgitate exact memorization.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

Mixture-of-Experts (MoE) models can regurgitate exact memorizations at 20B active parameters comparable to what previously required a 72B dense model.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

Published research and empirical benchmarking corroborate the claim that Mixture-of-Experts (MoE) models can achieve verbatim memorization and prefix-completion rates at roughly 20B active parameters comparable to those seen in ~70B–72B dense models. Specifically, Jelassi et al. (ICLR 2025) demonstrated that verbatim memorization capacity in MoEs scales primarily with total parameter capacity rather than active per-token parameters. Furthermore, frontier model evaluations between December 2025 and August 2026 (including the K2-V2 Technical Report and 'Memorization Diagnostics for Code LLMs') observed that sparse MoE models with low active parameter footprints match or exceed the pre-training extraction/memorization rates of dense baselines such as Llama 3.1 70B and Qwen 2.5 72B.

Jelassi et al. (ICLR 2025, 'Mixture of Parrots: Experts Improve Memorization More Than Reasoning') showed that MoE models scale memorization with total parameter count, allowing models with lower active compute to match the memorization capacity of dense models requiring substantially higher active parameters.
Empirical evaluations in late 2025 and 2026 (e.g., K2-V2 Technical Report and 'Memorization Diagnostics for Code LLMs') benchmarked verbatim extraction/retrieval rates on pre-training data, confirming that MoE models with low active parameter counts match or exceed 70B–72B dense baselines (such as Llama 3.1 70B and Qwen 2.5 72B).

Checked against

[1] paralleliq.ai [2] softwareseni.com [3] cornell.edu [4] imachines.com [5] neurips.cc [6] openreview.net [7] arxiv.org [8] harvard.edu

"Mixture of Parrots" "72B" OR "20B" OR "prefix" MoE memorization prefix completion active parameters 72B dense Pratyush Maini Mixture of Experts memorization 20B 72B "regurgitate" "memorization" "mixture of experts" "active parameters" "Mixture-of-Experts" memorization "active parameters" dense MoE vs dense memorization "prefix" "regurgitate" OR "extractable" "memorization" "20B" "72B" "MoE" "Pratyush Maini" "memorization" "Mixture-of-Experts" OR "MoE" "Mixture of Parrots" memorization "memorization" "mixture of experts" "dense" "72B" OR "70B" OR "20B" prefix completion

assessed as of 2026-09-02

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

Mixture of Experts

⚑ Spot an error? Report it

More from Pratyush Maini

Assertion Open · timeframe Feb 2027

Modern LLMs verbatim regurgitate JEE exam questions from two-word prompts

“We consistently saw how many of these, like, models today are being, like, massively, like, kind of fine-tuned on problems from... Like, oversight? Massively worked with. Like, even, like, imagine if I ask you the light bulb, what comes next in your mind? It w…”

Pratyush Maini Feb 10, 2026 ▶ 3:36 ⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Assertion Not checkable as stated

Labs train LLMs on benchmark questions for multiple epochs late in training

“It's very clear how the last stage of training for many of these models does have a massive amount of example or examine. Benchmaxing. Because the model will not, like, behaviorally complete exam questions with options if they have not really seen it at the en…”

Pratyush Maini Feb 10, 2026 ▶ 4:17 ⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Assertion Not checkable as stated

Self-reflection training data is now core to all frontier foundation models

“What this suggests about the GPT training data is that the self-reflection data has now actually become pretty much core to the training of all frontier models, because we're seeing that happen in non-instruct models across the board.”

Pratyush Maini Feb 10, 2026 ▶ 15:26 ⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Insight

Core AI capabilities must be built during pre-training, not just fine-tuned

“If there is a core capability that you actually care about, that capability should be part of the foundation and not a fine-tuned artifact.”

Pratyush Maini Feb 10, 2026 ▶ 18:53 ⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Prediction Not checkable as stated

Enterprises will widely adopt specialized AI pre-training in 2026 and 2027

“So I think like, 26 and 27 are going to be the years where different enterprises start doing specialized pre-training, because the cost of pre-training really amortizes itself very fast.”

Pratyush Maini Feb 10, 2026 ▶ 19:27 ⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Insight

Small specialized pre-trained models can match capabilities of larger fine-tuned models

“When you think of the fact that by doing specialized pre-training, you can train a smaller model, which is as capable as a much larger model when fine-tuned.”

Pratyush Maini Feb 10, 2026 ▶ 19:38 ⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

MoE models with 20B active parameters match 72B dense models in memorization

More from Pratyush Maini

Turn any episode into a week of clips.