StarZero All shows Try the podcast clipper →

SWE-bench Multimodal

part of SWE Bench

2 statements across 1 episodes · 0 bullish · 1 bearish · 1 people on the record · first statement Oct 19, 2024 by Jesse Hu · said 6 times in 4 episodes since 2024 · across every show →

Mentions by year

brought up most by Shawn Wang (1), Jesse Hu (1), Erik Schluntz (1)

tap a year for its mentions
00214220242025episodesmentions
01220242025episodes it came up in
00112220242025episodesmentions per episode
2025 4 mentions in 2 episodes 2 per episode
2024 2 mentions in 2 episodes 1 per episode

every mention, scene by scene, with the transcript →

Everything said about SWE-bench Multimodal, oldest first

Oct 19, 2024 negative
Opinion
Hu: Evaluating UI correctness in SWE-bench Multimodal is highly subjective
“Zooming out, if you're true to try to judge whether a UI is correct, it's like extremely subjective. It might be iterative. It might be, you know, I have to interact with it first to get it right. So I'm super curious to see how they actually do the judging cr…”
Jesse Hu Oct 19, 2024 ▶ 36:56 [Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu
Oct 19, 2024 neutral
Assertion Supported
SWE-bench Multimodal paper baseline scores 12 percent
“They want to show off that this is, you know, guys, this is really hard. It's a really hard benchmark, so we can only get 12% on it today. According to, you know, their implementation.”
Jesse Hu Oct 19, 2024 ▶ 34:48 [Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 200 episodes transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more