The Exchanges

Q So what I'm thinking as we start, just kind of give us like a brief history lesson of just like what happened in November and where are we today? What's possible now?

A Well, let's, let's talk about all of twenty-twenty-five very briefly. Um, twenty-twenty-five was the year that especially Anthropoc and OpenAI realized that code Is the application. Like being able, having these things generate code. I think partly because, um, Anthropic came up with Clawed Code back in sort of February of 2025, and it took off like crazy, and a bunch of people started signing up for 200 dollar a month accounts. And so suddenly, wow, it turns out people are willing to pay a lot of money for this stuff, for that specific field. Both Anthropic and OpenAI spent the whole of 2025 focusing all of their training efforts on coding. If you look at what they were doing, it was all the reinforcement learning stuff. The reasoning trick, the thing where the models say they're thinking, that was new in late 2024. Like OpenAI's O-One was the first model to exhibit that. And now all of the models do it. So that was the other big trend of last year was these reasoning models. Turns out reasoning is great for code. It can reason through code and figure out the root of bugs and all of that. And so the end result of this, the end result of these two labs throwing everything they had at making their models better at code Is in November, we had what I call the inflection point where GPT, 5.1 and Claude Opus 4.5 came along and they were both just, they were incrementally better than…

AI assessment note: “in November, we had what I call the inflection point where GPT, 5.1 and Claude Opus”

Q I wasn't planning to go here, but you're, you famously created the Pelican riding a bike benchmark for the quality of imagery. Yeah. Anything there that might be worth sharing?

A So this one's fascinating. About a year and a half ago, I started benchmark. So there are lots of benchmarks, these models, and they're all these numeric things like it scored 72% on terminal bench or whatever. And those always frustrated me because they don't really tell you anything interesting. Like if this one got 74 and this one got 72, Does that actually mean that one of them is better at something than the other? And so basically to make fun of the benchmarks, I started my own benchmark, which was generate an SVG of a pelican riding a bicycle. And it's an SVG. This isn't a test of the image models. This is a test of the text models because they can all output SVG code. And if you ask them to draw you an SVG of something, they're almost universally terrible because they don't have good spatial reasoning and like, Drawing things by plotting out vectors is difficult anyway. So I started getting the models to render, generate an SVG of a pelican on a bicycle, because then you can look at them. You can say, here's one, here's one model, here's the other, which is best. And the weirdest thing happened where there appears to be a very strong correlation between how good their drawing of a pelican riding a bicycle is and how good they are at everything else. And nobody can explain to me why that is. But as I started looking at these things, I realized, wow, The better models rea…

AI assessment note: “there appears to be a very strong correlation between how good their drawing”

Q And then how do you actually use this? Is this like, feed it into the LLM when you're building, or is it on occasion, go look at this, go look at that. Is that like in the memory or not?

A It's definitely both, but the key, the trick that I've been using lots is, especially for my little HTML and JavaScript tools, you can tell an LLM to consult them and combine them. So a very early example of that is, um, I'd written some code pre-LLMs, which used a PDF library from Mozilla. So it's in JavaScript, but it can open up a PDF and show you that PDF on the page. And I'd also written some code that used Tesseract, which is an OCR library that can run in your browser and do actually really good OCR all in JavaScript. And I just realized I wanted to do OCR against PDF files. So I told Claude opus three, I think back then I said, here is the code, like. Here's the code for the OCR, the PDF thing I did. Here's the code for the OCR thing, build a new thing that can open a PDF file and OCR every page. And it did it. And these days I'll often just tell Claude code, here's the paste in the URL to this thing, this thing here, here's another thing, go and read the source code and then solve this new problem. And it works so, so well. My research repository, I'll say things like, um, Check out Simon W slash research from GitHub and look at how, look at the ones in there that deal with WebAssembly and Rust, and then use that to feed into solving this new task in WebAssembly and Rust. Cause they, the, the, it's hard to overstate how good these things are with, if at reusing context…

AI assessment note: “It's definitely both, but the key, the trick that I've been using lots is”

Q So interesting. Okay. Final question. What are you, like, what are you up to? What's next for Simon? What should people know about what you're doing these days? What's coming next? Writing a book? Maybe building your claw?

A Yeah. So, I mean, my, my primary day, my day, my, my day job is open source tools for data journalism specifically. And I've been working on these for like More than five years now. And the idea is to build software that helps a journalist tell stories with data, which doesn't make you any money because journalists haven't got any money. But if I can help journalists tell stories with data, that's valuable to everyone else in the world with data that they need to interrogate. And what's been interesting over the past, especially over the past year is I've started bringing my interest in AI and my interest in journalism together. And it's like, okay, what are the things that I can build for journalists using AI? That can help them find stories and data, which given the AI makes things up and hallucinates and so forth, you would have thought that it's a very bad fit for journalism where the whole idea is to find the truth. But the flip side is journalists deal with untrustworthy sources all the time, right? The art of journalism is you talk to a bunch of people and some of them lie to you and you figure out what's true. So as long as the journalist treats the AI as yet another unreliable source, they're actually better equipped to work with AI than most other professions are. And so. I'm building things where you can like feed in PDFs of police reports, and it'll pull out the key…

AI assessment note: “I've started bringing my interest in AI and my interest in journalism together”

Q is the big now gap and opportunity, which is coming up with the idea. What the heck should we build? Because then once you tell the AI build this thing, as you're describing, it's getting better and better at building something great. Have you had any luck yet with using AI there? And do you think it starts to eat that and just becomes the strategy, you know, PM basically?

A So this is one of the most interesting problems we're having with all of this is we've taken the writing code bit and we've massively accelerated that. Now the bottlenecks are everywhere else, right? Like how do we redesign our processes now that the bit that used to take the longest, right? It used to be, you'd come up with a spec and you hand it to your engineering team. And three weeks later, if you're lucky, they'd come back with An implementation for you to then start. And now that, that maybe that takes three hours depending on how well established the coding agents after that kind of thing. So now what right now, where else are the bottlenecks? I don't think it's, I mean, that's coming with the initial ideas. Um, anyone who's done any product work knows that your initial ideas are always wrong. What matters is, is proving them, right? It's, it's, it's, it's testing them. We can test things so much faster now because we can build workable prototypes so much quicker. So there's an interesting thing I've been doing in my own work where any sort of feature that I want to design, I'll often prototype three different ways it could work. Cause that takes very little time. And then I can start experimenting them and trying them and seeing which ones I like. And that, that feels to me like the really transformational step here is that when you get AI involved in your ideation pha…

AI assessment note: “when you get AI involved in your ideation phase, it's much more about the prototypes”

Q back to that factory. I was talking to the founder of linear the other day and this idea of the factory, and we were just like, like a factory doesn't sound like a place that'll create amazing products. It feels like, you know, like what are the chances that'll create something beautiful and innovative? So either that's the wrong word or it's just, this will lead to bad stuff. Probably.

A I feel like the word artisanal does like, like artisanal to handcrafted software, I think is going to be valued more. Something I've noticed in my own work is Sometimes I'll have an idea through a piece of software Python library or whatever, and I can knock it out in like an hour. And get to a point where it's got documentation and tests and all of those things. And it looks like the kind of software that previous I just spent several weeks on, and I can stick it up on GitHub and everything. And yet I don't believe in it. And the reason I don't believe in it is that I got to rush through all of those things. I think the quality is probably good, but I haven't spent enough time with it to, to feel confident in that quality. Most importantly, I haven't used it yet. Like it turns out. When I'm using somebody else's software, the thing I care most about is I want them to have used it for, for months, right? I want other people to have put that software into practice. So I've got some very cool software that I built that I've never used. Like it was so, it was quicker to build it than to actually try and use it. And so the way I've been dealing with that, I always put alpha on it. Like if you see my software and it says it's an alpha, that probably means I haven't actually used it yet for most of my projects. Which is a bit of a cheat code, you know, um, alpha this, but isn't that …

AI assessment note: “artisanal to handcrafted software, I think is going to be valued more.”