The Exchanges

Q Yeah. Um, so, you know, I think the, this is interesting. Um, the, the, how do you compare it versus the other benchmarks that you see out there, like the suite bench. And then, you know, more recently now people are starting to compare benchmarks with The, like, like actual money-making projects, like Sui Lancer, something that OpenPlay recently released. Yeah, just any thoughts on the meta game of benchmarking?

A Yeah, I mean, I think, like, the This is, you know, I guess like in terms of experimental design, right? Like the Sweebench category of like starting from real GitHub data and like working towards that does feel like a really nice like end-to-end test. It's almost kind of like in my mind, you know, feels like the same type of test as like an integration test. This test is definitely a lot more targeted, right? It's like saying for this very specific skill of Taking a front end app and taking a specification of it. Can a agent then perform in this way that is easy to grade, right? It's like easy to say that given a front end app and given a back end implementation, like just, you know, test the functionality that it works because then also the front end is kept standard, right? So I think like, you know, the loss of generality here is in my mind, it's the trade off is that versus like, uh, power to grade, right? Like here we can have a, without having a gold standard of what the answer should be, we still can, uh, evaluate the solutions quite effectively. Right. Um, so I think like exploring different points on that trade off space of like generality versus like focused and Easy to grade is something that we're pretty interested in. We have another suite of evals for, um, we just call them convex evals that are more coding, like kind of small units of code focused, and they're j…

AI assessment note: “the Sweebench category of like starting from real GitHub data... feels like a really nice like end-to-end test”

Q just my, uh, I think we were talking about, and then if you, if you don't have anything in this feature, we can, we can, uh, stop it as well. But, uh, it's more just about like, uh, I guess Running Convex in, in the, in the age of AI, right? Like, like, I think this is something that you were saying in pre-chat about building developer tooling in general.

A Yeah, totally. So we've been at Convex since 20, 21 and, you know, a long, no time we've like, especially because we're kind of in this, like think from first principles, like redesign things that have been the same for a while. We've, you know, thought a lot about ergonomics for humans and like what helps people who may not have a lot of backend experience write really effective Convex code. And I think, you know, in the past year and certainly in the past few months, it's just been very apparent that You know, people are writing code really differently and, you know, if someone's a dev tool startup and they're writing something that is like in some set of APIs or systems that are targets for AI coding, they, you know, it's silly not to strongly consider how AI coders perform on one's platform. So that it was like the, you know, motivations for kind of investing in making some of this analysis a bit more rigorous. And then also, yeah, like, you know, users kept on telling us they're like, Hey, like Convex works really well for this, but I noticed that there's some knowledge gaps and there's like consistent hallucinations. And I think shaping one's platform to be a great coding target is like, you know, something that's just table stakes in 20, 25.

AI assessment note: “shaping one's platform to be a great coding target is like, you know, something that's just table stakes”

Q How, how do you do hints? I feel like if the model has been working for like 40 minutes, how do you catch up to where the model is? This is something I'm running more and more into when I use the cursor region. It's like, I just let it go, and then it's like, it doesn't work, but like, I don't even know what it did this old time.

A Totally. Yeah. It's, you know, this is why, you know, part I'll just, you know, in terms of transparency, right? Like why I cut off the first two tasks at 30 minutes is that it just takes a lot of human time to like, I can't just let it run in the background to, cause I'm trying to administer this test as fairly as possible. Right. And so like making sure that I'm giving good hints for all three of the backends and all three of the tasks. And yeah, like it is just, you know, noticing that the model's getting into a loop. So even though it might Think it's making progress. Like noticing that it got back to a state that it was at before stopping it and then doing some debugging and very quickly, like seeing like, this is the thing that, um, it should do differently. So I actually have some details here on like insight gave this, like, um, you know, for chat and the fast API, um, it was getting really confused about trying to combine some, um, tense that query stuff with the SSE and like, At a certain point, I just said like, hey, can you, you should just directly use use effect and here's what it should do. There's like another thing I got confused on where I noticed that it was like double stripping the data colon prefix for it, and it just couldn't figure it out. It kept on like alternating between the two. So yeah, I think this is like part of the, you know, for like the meta …

AI assessment note: “noticing that the model's getting into a loop... stopping it and then doing some debugging”

Q the code, you have the environment and all of that. What are like some of the design decisions that you guys made here on what to expose to the model? It's like, you're not bringing people to code somewhere else. You're just letting them do this in their existing IDE. Did you ever think about building your own kind of like only one environment for people to work in or?

A Yeah, I think some of it is, um, you know, for this particular study, it's very manual still. And so I think like I was just most familiar with cursor composer. So I like, we have a standardized environment where we create the front end app and next, and we set up authentication and we set up some user tables and stuff like that. And then we, from that starting point, kind of let cursor composer go off. I think, you know, it's like kind of that future work of, um, I would love to just be able to use like cloud code, use, um, or even just make stuff more standardized where it's like running and, you know, it's not just running on my laptop, it's running on some, a containerized environment I can do at scale.

AI assessment note: “for this particular study, it's very manual still. And so I think like I was just most familiar with cursor composer.”

Q I was gonna say, do you have a sense of how people can then take these hints and try to work them back into the model, maybe put it in the cursor rules, any research you've done on that side?

A That's a really good question. I think the main angle I was thinking of is like looking at these hints and there's some kind of like stuff in the observation section on What things the model got stuck on and what that implies for like, what makes something easy to generate? Like one example where convex didn't do super well is this hint at 48 minutes. Convex has like a pretty subtle distinction between null and undefined, like just similar to JavaScript. And we noticed that like, because this is a subtle thing that's unexpected, it was The model got stuck on it and couldn't even figure it out. And so this is something where I had to explicitly tell it like, Hey, as a convex expert, this is the thing that you are running into. So, you know, from a platform perspective, like I would, if I, you know, if, if we can do it in a way that's backwards compatible, I would love to tweak convex to just match the model's expectations here. But another approach would be to, uh, tweak the, like the guidelines that we prompted with.

AI assessment note: “another approach would be to, uh, tweak the, like the guidelines that we prompted with.”

Q cool demo. People love seeing it. Like then they're not like, okay, I like, but how do I use this? So basically I think my question is like, how does Convex see like the backend, like being the backend of agents? Do you have everything you need? Is it, are you just waiting for the market to catch up or like what, where are we in, in, in that cycle?

A Yeah. I think it's like a, it's very interesting, like strategic question. Right. And that as a backend platform, um, you know, I think when we were really starting to dig into like, you know, this particular round of AI investment from us, like, I think it felt like that there's kind of two paths in the decision tree. There was like people using AI coding to write backends that are like kind of the same as they were before. There's like, you know, there's like The hypothesis there being that there's all this pent up demand for apps that maybe aren't so different than apps a year ago or two years ago, and that like a much larger set of people can do it at a much smaller cost. And then there's the other side of like, will back-end infrastructure look different with all the changes in AI? So two years ago, that was like vector databases, right? Which like, you know, were used before, but now are just so ubiquitous in a way that weren't. There's like, I wish I remembered off the top of my head. There's a really good blog post on, um, agent architectures around like, you know, using temporal, right. And like using, you know, like the usual techniques for making like a unreliable service and adding retries and extension backups and all of that. I think like, yeah, we're very interested. I mean, we have like a workflow system on convex, like for durable execution. And I think we're s…

AI assessment note: “we're still like investing more on that first category of like saying like”

Q Fair enough. So what were your results? What's interesting?

A Yeah. So I think like, you know, the thing that's the kind of way that we measure Performance here is that, you know, we put a time limit on each, you know, for each element of the three by three matrix. And for the first two tasks, we said, 30 minutes and then 60 minutes for the files task. And then along the way, um, this is, you know, currently still really manual. Cause I think it's kind of, you know, it's still, we're not exactly sure how to do evals for this, like thing where the human needs to be in the loop. So along the way I am. Yeah. Giving it hints if needed, if it gets the, a cursor composer gets stuck and it gets into loops. And then I'm also, if it thinks it's done, I'm going and like trying the app and starting grading and then reporting any errors that, uh, you know, show up in the browser and aren't immediately visible. So I think, so then this is the kind of quantitative data. Oh, and then the final step, sorry, is that we, once either the time limit is done or. The model stops making progress and it, uh, then we grade it. So we say, for each particular task, there's like, you know, if you have a chat app, like does posting a message work? Does posting a message work and live update on the page about a page refresh? You know, if you can navigate to another channel, does it show that channel's messages? So we have kind of some predetermined rubrics for each on…

AI assessment note: “the kind of way that we measure Performance here is that”