Mar 19, 2025 · 31m · latent-space
Fullstack-Bench: The Eval for Coding Agents — with Sujay Jayakar, Chief Scientist, Convex
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of Latent Space, Convex Co-founder and Chief Scientist Sujay Jayakar introduces Fullstack-Bench, a comprehensive benchmark evaluating AI coding agents across different backend architectures. He discusses model failure modes, the evolution from Developer Experience to Agent Experience, and the infrastructure required to support autonomous software development.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 4.2% of the talking time here. How this is scored →
The hosts as informed peer 5.0 Guest teaching 3.9 Guest disagreement 1.1 The hosts pushing back 1.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 23:20 Pushing past the assumption that DX equals AX
Sujay rejects the conventional wisdom that developer ergonomics naturally translate to LLM usability, noting models get tripped up by subtle deviations from legacy frameworks like Firebase.
Hardest push from the hosts ▶ 24:06 swyx argues AI locks in legacy APIsShawn pushes back on API flexibility, arguing that LLM training data effectively calcifies API design and makes switching syntax nearly impossible.
Biggest teaching moment ▶ 18:30 Sujay details formal concurrency verification for agentsSujay educates the audience and hosts on applying distributed systems tooling like Jepsen and Elle to automatically grade agent-generated database code under high concurrency.
The host holds their own ▶ 5:11 swyx contextualizes benchmark with 7GUIs and RealWorldShawn demonstrates deep familiarity with UI/backend evaluation literature by connecting Fullstack-Bench to historical projects like 7GUIs and Thinkster's RealWorld spec.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Fullstack-Bench Experimental Setup and Benchmark Design | 6 | 3 | 1 | 2 | Shawn Wang brings up existing benchmark comparisons like 7GUIs and Thinkster RealWorld app suites to contextualize Fullstack-Bench. Sujay readily agrees with the limitations regarding templates and scaffolds. | |
| Benchmark Results and Human-in-the-Loop Interventions | 5 | 4 | 1 | 2 | Alessio queries how human intervention and hint-giving works when models run off-track for long periods. Sujay explains the practical difficulties of manual eval administration and debugging loops. | |
| Model Discrepancies, Cursor Rules, and SWE-bench Comparison | 6 | 4 | 2 | 2 | Alessio shares practical experience using Cursor rules to stop models defaulting to deprecated RSpec matchers, while Sujay discusses Claude 3.7 regressing compared to 3.5 on internal Convex evals. | |
| Future Benchmarking: Concurrency, Verification, and Throughput | 3 | 6 | 0 | 0 | Sujay outlines advanced verification plans using Jepsen model checking for serializability and TPC-style throughput benchmarks, with the hosts purely listening. | |
| Developer Experience vs. Agent Experience and API Calcification | 6 | 3 | 2 | 4 | Shawn challenges the assumption that good DX maps directly to agent experience (AX) and posits that pre-training cutoffs calcify API design. Sujay agrees that models struggle with subtle differences from Firebase. | |
| Agent Architectures, Durable Execution, and AI Town | 5 | 3 | 1 | 2 | Shawn asks why the industry saw a lack of practical follow-through after the viral AI Town demo. Sujay explains the distinction between traditional apps built via AI and fundamentally new agent backend architectures. | |
| Key Takeaways for AI Tool Builders and Podcast Conclusion | 4 | 4 | 1 | 0 | Sujay wraps up with key lessons from the benchmark, contrasting procedural debugging with declarative pitfalls like Supabase RLS infinite recursion before concluding the episode. |
Statements from this episode (11)
Claude 3.5 in Cursor autonomously codes for 10 plus minutes
“When it has the right feedback in cursor composer, and this was even on cloud three, five, it can just autonomously code for a 10 plus minutes and it can fix its own bugs. It can get to the point where it's like pretty much a fully working app with just an ini…”
Cursor Composer solves Convex benchmarks but fails on alternative backends
“We did notice that I mean, with convex, it pretty much autonomously just solves the first two tasks. It has a few round trips on like some errors that are only show up and playing with the front end. And then it's able to complete this files task and kind of g…”
Jayakar: AI Models Stalled on Convex's Subtle Distinction Between Null and Undefined
“Convex has like a pretty subtle distinction between null and undefined, like just similar to JavaScript. And we noticed that like, because this is a subtle thing that's unexpected, it was The model got stuck on it and couldn't even figure it out. And so this i…”
Claude 3.7 performed worse than Claude 3.5 on Convex evals
“For example, we just tried clod three seven and it performs worse than clod three five on convex evals with the same prompting.”
OpenAI's o3 outperforms GPT-4o on Convex evals by a small margin
“You know, oh, three does do better than four. Oh, I mean, we use brain trust for tracking all this quantitatively, but I can't remember off the top of my head, but it's not like a slam dunk.”
Jayakar prototypes AI coding benchmark using Jepsen's Elle model checker
“I think like, you know, I've wired up a version of this, you know, using L, which is the like model checker from Jepson and, you know, can a model write highly correct, highly like code that executes under very high concurrency, even when there's”
AI hallucinates Convex code due to API similarities with Firebase
“I mean, I think one thing that's kind of interesting given these, you know, models inherent internal structure is that we noticed that a lot of hallucinations come from parts of our API that are like very close to Firebase, but not exactly Firebase.”
Swyx: AI pre-training datasets calcify software APIs against future changes
“Like one tricky thing, I feel like it's almost like AI calcifies your API. The cost of switching APIs is so high because it's, you don't know what the hell is in those, the data set. You're not going to reach out to every model lab and like ask them to update …”
End-to-end type safety and tight feedback loops improve AI agent performance
“It's like, you know, having just really tight feedback loops and having strong guardrails, like in convex, that's like end to end type safety, but that's like, you know, could take it in many, many different forms, right? Like making it so code is very cheap a…”
AI models struggle debugging Supabase RLS recursion compared to procedural code
“The particular example was like RLS rules and Supabase where debugging like an infinite loop for infinite recursion for the RLS rules was something that the models just really struggled with in a way that we didn't see for procedural code.”
Strong library abstractions prevent AI models from breaking backend wiring
“Models, you know, when given the full flexibility of fast API and doing SSE and wiring everything from scratch. It just couldn't help, but messed it up. So having like kind of strong life, like choosing good libraries that have strong abstractions is like, and…”