The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 10 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Cursor Composer solves Convex benchmarks but fails on alternative backends
“We did notice that I mean, with convex, it pretty much autonomously just solves the first two tasks. It has a few round trips on like some errors that are only show up and playing with the front end. And then it's able to complete this files task and kind of g…”
Claude 3.7 performed worse than Claude 3.5 on Convex evals
“For example, we just tried clod three seven and it performs worse than clod three five on convex evals with the same prompting.”
AI models struggle debugging Supabase RLS recursion compared to procedural code
“The particular example was like RLS rules and Supabase where debugging like an infinite loop for infinite recursion for the RLS rules was something that the models just really struggled with in a way that we didn't see for procedural code.”
AI hallucinates Convex code due to API similarities with Firebase
“I mean, I think one thing that's kind of interesting given these, you know, models inherent internal structure is that we noticed that a lot of hallucinations come from parts of our API that are like very close to Firebase, but not exactly Firebase.”
Claude 3.5 in Cursor autonomously codes for 10 plus minutes
“When it has the right feedback in cursor composer, and this was even on cloud three, five, it can just autonomously code for a 10 plus minutes and it can fix its own bugs. It can get to the point where it's like pretty much a fully working app with just an ini…”
OpenAI's o3 outperforms GPT-4o on Convex evals by a small margin
“You know, oh, three does do better than four. Oh, I mean, we use brain trust for tracking all this quantitatively, but I can't remember off the top of my head, but it's not like a slam dunk.”
End-to-end type safety and tight feedback loops improve AI agent performance
“It's like, you know, having just really tight feedback loops and having strong guardrails, like in convex, that's like end to end type safety, but that's like, you know, could take it in many, many different forms, right? Like making it so code is very cheap a…”
Strong library abstractions prevent AI models from breaking backend wiring
“Models, you know, when given the full flexibility of fast API and doing SSE and wiring everything from scratch. It just couldn't help, but messed it up. So having like kind of strong life, like choosing good libraries that have strong abstractions is like, and…”
Jayakar: AI Models Stalled on Convex's Subtle Distinction Between Null and Undefined
“Convex has like a pretty subtle distinction between null and undefined, like just similar to JavaScript. And we noticed that like, because this is a subtle thing that's unexpected, it was The model got stuck on it and couldn't even figure it out. And so this i…”
Jayakar prototypes AI coding benchmark using Jepsen's Elle model checker
“I think like, you know, I've wired up a version of this, you know, using L, which is the like model checker from Jepson and, you know, can a model write highly correct, highly like code that executes under very high concurrency, even when there's”