Artificial Analysis
includes Artificial Analysis Intelligence Index, Artificial Analysis Quality Index
20 statements across 4 episodes · 8 bullish · 2 bearish · 6 people on the record · first statement Aug 4, 2025 by Stefano Ermon · said 77 times in 17 episodes since 2024 · across every show →
Mentions by year, the whole family
brought up most by Shawn Wang (31), George Cameron (16), Micah Hill-Smith (14), Andrew Feldman (4), Stefano Ermon (2), Lin Qiao (2), Pranav Reddy (1), Philip Kiely (1)
tap a year for its mentions
2026 47 mentions in 4 episodes 12 per episode
- Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith
- The AI Frontier: from open weights to open research — Eiso Kant, Poolside AI
- Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten
- The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin
- every mention in 2026, scene by scene →
2025 24 mentions in 8 episodes 3 per episode
- [State of Evals] LMArena's $1.7B Vision — Anastasios Angelopoulos, LMArena
- ⚡️Raising $1.1b to build the fastest LLM Chips on Earth — Andrew Feldman, Cerebras
- 2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents
- ⚡️Mercury: Ultra-Fast Diffusion LLMs — Estefano Ermon, CEO Inception Labs
- The Great Evals Debate — Ankur Goyal & Malte Ubl
- Building Jamba 3B: the tiny Hybrid Transformer State Space Reasoning Model - Barak Lenz, CTO of AI21
- The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)
- AI Engineering for Art - with comfyanonymous
- every mention in 2025, scene by scene →
2024 6 mentions in 5 episodes 1 per episode
- Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI
- The State of AI Startups in 2024 [LS Live @ NeurIPS]
- [LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models
- Building an open AI company - with Ce and Vipul of Together AI
- The Four Wars of the AI Stack - Dec 2023 Recap
- every mention in 2024, scene by scene →
Everything said about Artificial Analysis, oldest first
Aug 4, 2025 bullish
Inception generalist model matches Claude Haiku quality at 5-10x speed
“We had our generalist model evaluated by artificial analysis and the intelligence score from AA artificial analysis around 40. So it's comparable to GPT, 4.1 nano, cloud haiku, kind of like Close source speed optimized models. It's roughly comparable in terms …”
Oct 1, 2025 bullish
Dec 31, 2025 neutral
Dec 31, 2025 positive
Arena's organic user prompts provide realism that Artificial Analysis lacks
“They have arenas, but the arenas are not based on organic usage. Like the thing that distinguishes our platform versus theirs is that the users are actually inputting their own use case. They're actually asking their own question. And that gives a level of rea…”
Jan 9, 2026 neutral
Cameron: Model performance correlates with total parameters, not active parameters
“We, in our benchmark, see a lot of performance correlated more with total parameters than active, and not that correlated with how sparse like the models are. Our accuracy benchmark is part of a omniscience. It's very correlated with total. It's not correlated…”
Jan 9, 2026
Jan 9, 2026
Artificial Analysis Intelligence Index synthesizes 10 evaluation datasets
“The artificial analysis intelligence index, which is our synthesis metric that we pulled together currently from 10 different eval data sets to give what? We're pretty confident is the best single number to look at for how smart the models are.”
Jan 9, 2026 neutral
Artificial Analysis uses mystery shopper accounts to prevent endpoint manipulation
“We have what we call a mystery shopper policy, and so, and we're totally transparent with all the labs we work with about this, that we will register accounts not on our own domain and run both intelligence evals and performance benchmarks without them being u…”
Jan 9, 2026 negative
Jan 9, 2026 positive
AI2's OLMo 3 32B leads Artificial Analysis's 18-point Openness Index
“It's out of 18 currently. And so we've got an openness index page, but essentially these are points. You get points for being more open across these different categories and the maximum you can achieve is 18. So AI two with their extremely open OMO three, 32 B…”
Jan 9, 2026
Jan 9, 2026 bullish
Models perform better in custom agent harnesses than native web chatbots
“And what's really interesting is that if you compare, for instance, Claude, 4.5 Opus using the Claude web chatbot, it performs worse than the model in our Agentic harness. And so in every case, the model performs better in our agentic harness than its web chat…”
Jan 9, 2026 positive
Jan 9, 2026 neutral
Reasoning models consume 10x more tokens on average than non-reasoning models
“So, earlier this year, and probably when you and George last spoke for the AI engineers world's fair, we had this great slide that was super easy, where we would show that the average reasoning model is using 10 times the number of tokens per query in our inte…”
Jan 9, 2026 positive
Jan 9, 2026 neutral
Cameron: General model intelligence does not correlate with hallucination rates
“One interesting aspect is that we've found that there's not really a, not a strong correlation between intelligence and hallucination rate. That's to say that the smarter the models are in a generalist sense isn't correlated with their ability to, when they do…”
Jan 9, 2026 positive
Jan 9, 2026 neutral
Hill-Smith estimates Gemini 3 Pro parameter count hits 5 to 10 trillion
“You might reasonably form a view that there's a pretty good chance that Gemini three pro is bigger than that, that it could be in the five to 10 trillion parameter range. To be clear, I have absolutely no idea, but just based on this chart, like that's where y…”
Jan 9, 2026 neutral