METR, every mention
17 scenes, the whole family · ← back to METR
tap a year for its mentions
every year anyone Joel Becker 15Shawn Wang 13Alessio Fanelli 5Kyle Kranen 2
Verbatim, from the transcripts: the passages where METR comes up
Agent Inference at the "Speed of Light" — How NVIDIA moves like a $4.3 Trillion Startup
- ▶ 1:18:49 Kyle Kranen It's like, it's like the, the Meta index, right, is the thing that tracks like how long models are able to run. 2 times in the scene
Measuring Exponential Trends Rising (in AI) — Joel Becker, METR
- ▶ 0:00 Joel Becker So METR stands for M-E-T-R.
- ▶ 0:48 Shawn Wang We're back in the studio with Joel Becker from Meter. 4 times in the scene
- ▶ 4:28 Joel Becker One way is there's this, um, PowerPoint, internal meter PowerPoint from twenty-twenty-three, where we're trying to lay out our ambitions for, for, for what meter research might look like in the future, and there's this graph. 6 times in the scene
- ▶ 9:25 Shawn Wang Uh, yeah, and, but like, uh, hcast was like, it was like the first time I come across it, and, and the others, I, I, I mean, swa, is that, is that the meter proprietary ones?
- ▶ 13:15 Joel Becker You know, I think, um, meters out there trying to, um, support civil society, trying, trying to, um, provide high quality independent information to the public.
- ▶ 14:16 Joel Becker I think in some ways, like meter time horizon is,
- ▶ 23:19 Shawn Wang I think like basically no one is stopping to do science except for you guys, uh, because, uh, we know RCTs are the best, right? 2 times in the scene
- ▶ 25:21 Joel Becker Meta came out of ARC, so I think, um. 2 times in the scene
- ▶ 30:54 Joel Becker I, I'm, you know, I'm not speaking for everyone at Meta. 3 times in the scene
- ▶ 48:32 Joel Becker I think it's important that, um, you know, meet our employees not be, not be, um, uh, not be making bets on prediction markets like that.
- ▶ 49:48 Alessio Fanelli What are other interesting model evaluation trajectories or like anything that you're not doing at Meter that you've maybe seen other people do that you find interesting 2 times in the scene
- ▶ 56:19 Alessio Fanelli Like, I don't know if how you build the harness in meter or what or not, you always pick the best harness or compare them. 3 times in the scene
- ▶ 59:46 Shawn Wang Just to wrap up, like, I guess, what do we expect out of Meter in 26? 6 times in the scene
The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals
- ▶ 21:43 unnamed speaker I think the only other sort of major player in this field is meter, which has done the sort of long graphs. 2 times in the scene