Artificial Analysis, every mention
33 scenes, the whole family · ← back to Artificial Analysis
tap a year for its mentions
every year anyone Shawn Wang 31George Cameron 16Micah Hill-Smith 14Andrew Feldman 4Stefano Ermon 2Lin Qiao 2Pranav Reddy 1Philip Kiely 1Barak Lenz 1
Verbatim, from the transcripts: the passages where Artificial Analysis comes up
Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten
- ▶ 38:14 Philip Kiely Um, so the spreads that you see, if you like go on artificial analysis or you go on open router and you look at, you know, the, the worst provider to the best provider oftentimes can, can hit that kind of range.
The AI Frontier: from open weights to open research — Eiso Kant, Poolside AI
- ▶ 1:07:39 unnamed speaker Um, and so, I mean, the, the other way is just to give it to artificial analysis and let them run it like third parties. 2 times in the scene
The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin
- ▶ 28:26 Shawn Wang Uh, I do think like there are a bunch of like artificial analysis is obviously, uh, doing super well, uh, with, with their stuff.
Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith
- ▶ 0:09 Micah Hill-Smith Um, cause the like first time artificial analysis got mentioned on a podcast was you and Alessio on Land and Space. 2 times in the scene
- ▶ 4:16 Shawn Wang Um, what was the impetus for starting artificial analysis in the first place? 4 times in the scene
- ▶ 7:49 Shawn Wang The way that if I were to start artificial analysis at the same time you guys started, I would have used the Luther AI's, um, eval framework, uh, harness.
- ▶ 11:58 Micah Hill-Smith So like one of the things that we do is run an enormous number of all of our evals when we're developing new ones and doing upgrades to our intelligence index to bring in new things. 3 times in the scene
- ▶ 17:20 George Cameron Uh, and they've been great mentors to us as we've built artificial analysis. 4 times in the scene
- ▶ 19:13 Micah Hill-Smith So first, just like background, like we're talking about the artificial analysis intelligence index, which is our synthesis metric that we pulled together currently from 10 different eval data sets to give what? 3 times in the scene
- ▶ 22:23 George Cameron Currently, the Intelligence Index and evals focus a lot on kind of raw intelligence, but we kind of want to diversify how we think about intelligence, and we can, we can talk about it, but kind of new evals that we've kind of built and… 6 times in the scene
- ▶ 25:01 George Cameron But, um, something that I actually don't think every artificial analysis user knows is that you can customize our charts and, and choose what, uh, what models are highlighted.
- ▶ 26:55 Shawn Wang so this is the, the rough, uh, pitch of AAQI, uh, or is it AAQI or AAII? 2 times in the scene
- ▶ 33:14 George Cameron One eval that we added to artificial analysis is, is, is critical point, and it's really hard, uh, 2 times in the scene
- ▶ 51:10 George Cameron Yeah, we've looked at it from a evals perspective, and we love Terminal Bench and, and host benchmarks of Terminal Bench on artificial analysis.
- ▶ 52:11 Micah Hill-Smith Like long context reasoning data said last year, we did open source. 3 times in the scene
- ▶ 58:35 Micah Hill-Smith A couple of like the big trends that we track at artificial analysis over time and that like, we're always showing charts of on the trends page and these reports and stuff. 3 times in the scene
- ▶ 1:14:16 George Cameron And I'm not sure that that's, that's the case, but I think for artificial analysis, we'll keep benchmarking, uh, raw intelligence, but we also want to think about it and explore models more deeply across other axes as well. 2 times in the scene
- ▶ 1:14:52 Shawn Wang So what is going to be included in, let's say like a V three of the intelligence index, because obviously you're going to saturate. 6 times in the scene
[State of Evals] LMArena's $1.7B Vision — Anastasios Angelopoulos, LMArena
- ▶ 6:02 Shawn Wang Um, you're not the only player, uh, there are this artificial analysis started in the arena. 10 times in the scene
The Great Evals Debate — Ankur Goyal & Malte Ubl
- ▶ 27:51 Shawn Wang Uh, I think that's why like companies like Artificial Analysis, uh, LM Arena, these are, these are like the independents, like the new gardeners of like independent reading agencies for various companies.
Building Jamba 3B: the tiny Hybrid Transformer State Space Reasoning Model - Barak Lenz, CTO of AI21
- ▶ 12:20 Barak Lenz I think that that's the biggest feature because if you look, let's say it artificial analysis, many models out there, none of them is, is hybrid.
⚡️Raising $1.1b to build the fastest LLM Chips on Earth — Andrew Feldman, Cerebras
- ▶ 5:38 unnamed speaker You talk with, um, artificial analysis, or I'm familiar with them too.
- ▶ 11:03 Andrew Feldman I think also just go up and look at artificial analysis. 4 times in the scene
⚡️Mercury: Ultra-Fast Diffusion LLMs — Estefano Ermon, CEO Inception Labs
- ▶ 16:55 Stefano Ermon We had our generalist model evaluated by artificial analysis and the intelligence score from AA artificial analysis around 40. 2 times in the scene
The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)
- ▶ 14:31 Shawn Wang Artificial analysis also has one, which I think is kind of cool.
AI Engineering for Art - with comfyanonymous
- ▶ 15:39 Shawn Wang They just subscribe to Fofur AI and just see like, you know, what Fofur is doing.
2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents
- ▶ 14:18 Shawn Wang I would say that, you know, it's good to have an open model that is that big, that does well in those metrics, but anyone putting four or five B in production will tell you if you scroll down a little bit to the artificial analysis…
- ▶ 49:33 Shawn Wang I will also shout out Recraft has come out on top of the image arena that artificial analysis has done, has apparently taken Flux's place. 2 times in the scene
The State of AI Startups in 2024 [LS Live @ NeurIPS]
- ▶ 7:10 Pranav Reddy This is from artificial analysis, which tracks cost per token across a variety of different APIs and public inference options, and, like, we were doing some math on this.
Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI
[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models
- ▶ 1:05:58 unnamed speaker Have you seen an artificially analysis?
Building an open AI company - with Ce and Vipul of Together AI
- ▶ 47:08 Shawn Wang Yeah, I'll point people to artificialanalysis.ai, uh, which is a new one that
The Four Wars of the AI Stack - Dec 2023 Recap
- ▶ 26:00 unnamed speaker Um, so that, so, um, actually there was a really good website that was released, uh, just this week, um, called Artificial Analysis.