Oct 25, 2023 · 44m · mad
Diffblue’s AI Testing Paradigm Shift — CEO Mathew Lodge Explains How Code Writes Itself
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
On The MAD Podcast, Diffblue CEO Mathew Lodge explains how Reinforcement Learning enables autonomous unit test generation, revolutionizing enterprise legacy software modernization and developer productivity beyond traditional Large Language Models.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 24.1% of the talking time here. How this is scored →
Matt as informed peer 3.9 Guest teaching 3.4 Guest disagreement 1.6 Matt pushing back 1.6
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 33:52 Dismissal of Prompt Engineering
The guest forcefully rejects the term 'prompt engineering', arguing that applying traditional engineering logic to unpredictable LLMs is a complete fallacy.
Hardest push from Matt ▶ 6:53 Host halts premature deep diveThe host firmly cuts off the guest mid-sentence to hold off on unit testing details and keep the interview structure on track.
Biggest teaching moment ▶ 3:20 Search spaces in Chess vs GoThe guest educates the host on reinforcement learning fundamentals by contrasting chess brute-force search with Go's massive search space requiring probabilistic predictions.
Matt holds his own ▶ 23:44 COBOL legacy infrastructure contextThe host demonstrates deep knowledge of enterprise tech debt by citing COBOL's persistent dominance in global banking backends.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Opening Titles and FirstMark Branding Sequence | 3 | 5 | 1 | 3 | The host asks a broad intro question, allowing the guest to deliver an extended tutorial on LLM predictive completion vs Reinforcement Learning search spaces. The host interjects briefly to steer interview structure before asking a solid technical follow-up about RLHF and hybrid model ensembles. | |
| Understanding Unit Testing and Legacy QA Challenges | 2 | 4 | 1 | 1 | The host deliberately asks basic ELI5 questions to clarify unit testing concepts for a general audience. The guest walks through shift-left practices, legacy QA bottlenecks, and the friction of developer testing. | |
| Autonomous Test Generation and Compute Efficiency | 4 | 3 | 1 | 1 | The host demonstrates good tech intuition by mapping call graphs to visual mapping exercises and inquiring about CPU vs GPU compute overhead. The guest explains Diffblue's iterative test generation process. | |
| Enterprise Adoption, COBOL Migration, and Commercial Strategy | 5 | 2 | 1 | 1 | The host displays strong domain knowledge by bringing up COBOL migration challenges in financial infrastructure and enterprise software buyer personas. The exchange is highly collaborative as the guest outlines Diffblue's enterprise GTM. | |
| Developer Experience, AI Autonomy, and Impact on Software Jobs | 4 | 4 | 2 | 2 | The host raises potential developer friction regarding autonomous AI and quotes a Citibank executive regarding job impact. The guest reframes autonomy using a historical analogy of assembly language transitioning to compiled high-level languages. | |
| Industry Perspectives: Prompt Engineering, Model Size, and Open Source | 5 | 4 | 3 | 2 | The host engages with guest's public Twitter commentary regarding AI trends. The guest delivers strong contrarian views, dismissing prompt engineering as trial-and-error and explaining why smaller open-source models will win. | |
| The UK AI Startup Ecosystem and Global Talent | 4 | 2 | 2 | 1 | The host highlights notable UK tech startups, leading into a discussion on talent pools. The guest offers a candid critique of Brexit's impact on European tech talent acquisition. |
Statements from this episode (12)
Lodge: DeepMind shaved 7% off C++ sort algorithm using reinforcement learning
“Google DeepMind has also applied this approach Do things like code optimization, where essentially they're searching for more efficient implementations of a particular algorithm. So they shaved about seven percent off the sort algorithm in the C++ library, whi…”
Lodge: Diffblue uses ensemble of reinforcement learning and LLMs
“Yeah, we, so we use an ensemble approach. So we've got different aspects of the models and we've obviously been playing with large language models to see if we can improve on that process. So some of this is like, how do you make predictions? And you can use l…”
Mathew Lodge: Developers dislike unit tests because incentives favor feature delivery
“Fundamentally they're not paid to write tests. They're paid to deliver the functionality, the new things in the application that fixes the application, the updates and tests are just there to, as part of the process to help them.”
Diffblue generated 3,100 unit tests in eight hours for a bank
“You know, for one of the banks, they had an application. We wrote like 3100 tests in about eight hours. Now they estimated that that's like a year's work for a single developer, 3000 tests. And we can do that in eight hours.”
Private equity mainframe software acquisitions and price hikes drove companies to Java
“Because PE companies bought all of those software products and raised the price.”
Lodge: AI code generation mirrors the shift from assembly to compiled languages
“The analogy I like to make is, it's kind of like, ah, the shift, you know, 30, ah, 35 years ago from sort of assembly language to compiled languages, right? So what you did as a developer really changed”
Interactive AI code suggestions are strictly bottlenecked by human review speed
“You could really still only go as fast as a human can go, because a human has to review that, decide whether it's useful or not, if it is useful, make it, fit it into the code and finish the work. You're still basically going at the speed of a human, whereas w…”
Mathew Lodge argues prompt engineering is a fallacy and merely trial-and-error
“And so the idea that there's something called prompt engineering, engineering in the sense of you're applying an approach because it's going to give you the output that you desire. Is a complete fallacy. At best it's trial and error. It's prompt trial and erro…”
Diffblue generates unit tests in 1.5 seconds versus 40 seconds for LLMs
“We built a version of our product where we use the large language model to generate all the tests. So we took our reinforcement lending engine out and put in large language model, and we did a lot of work on prompts and so on. And you know, our product will wr…”
The AI market will shift toward smaller models running without GPUs
“It's one of the reasons I think that we're going to see more smaller models and you've got startups now emerging, which are essentially trying to, like, distill down, like from a large model into something much smaller that will run without a GPU, and they're …”