The Exchanges

Q or is that a combination of reinforcement learning and, and LLMs, or is that, um, you know, people, um, I'm sure I've heard, uh, of, uh, you know, GPT, uh, being, um, uh, having a, a, uh, sort of reinforcement learning from human feedback are, RLHF being a part of what they do overall. So is that, is that a hybrid approach or is that a solely, um, reinforcement learning?

A Yeah, we, so we use an ensemble approach. So we've got different aspects of the models and we've obviously been playing with large language models, um, to see if we can improve on that process. So some, some of this is like, how do you make predictions? Um, and you can use large language models to do some of that work. You can have them make some predictions for you about what might a good test look like and so on. Uh, so we, we do use a number of different techniques. It's not purely, um, uh, all about reinforcement learning, but the, the, the big difference is that in large language models, essentially the reinforcement learning is used to tune the model, um, and, uh, how to prefer some responses over others. And so part of what you're doing is correcting for the randomness that's in large language model, um, and, uh, teaching it to prefer some outcomes over others, um, which is, and fundamentally what we're doing with reinforcement was reinforcement learning first and foremost, how you guide the algorithm, how you assess, um, uh, the, uh, effectiveness of what we're doing, how we make predictions, That is ensemble, but it's, the emphasis is different, right? It's primarily about reinforcement learning as opposed to using reinforcement learning to correct what a large language model has produced.

AI assessment note: “we use an ensemble approach. So we've got different aspects of the models”

Q know, just looking around on, um, on Twitter and some press and everything you have, you have, um, you have some really interesting, uh, you know, spicy takes, I guess this is the term of the day, uh, spicy takes. Um, and, uh, one that caught my attention recently was, um, prompt engineering is not a thing. Uh, do you want to expand on that and explain what that is?

A Yeah. Yes, um, it was, uh, this came out some reflection about exactly how large language models work and, uh, and do what they do so well. And fundamentally, uh, and some, um, experiences using a very early versions of, uh, large language models for co-completion. Where we found is that very, very small changes in, um, what you asked the model to do and the code you asked it to write could result in huge differences in what you got, the output you got. And, and so, and that's, that's essentially what people are talking about with prompt engineering, because what, what Copilot is doing is basically doing, it's doing a lot of behind the scenes, doing a lot of work on the prompt to try and get you the best code, uh, for a particular, uh, situation. But fundamentally, these models are so big and so complicated that nobody knows what they're going to do. There's no, you don't have any predictive ability. And so the idea that there's, there's something called prompt engineering, engineering in the sense of you're applying an approach because it's going to give you the output that you desire. Is a complete fallacy. Um, at best it's trial and error. It's prompt trial and error. And there are rules of thumb about how you construct a good prompt and so on, which is, you know, based on the way that the, the backend model works, but it's not engineering. It's not engineering in the tradit…

AI assessment note: “The idea that there's something called prompt engineering is a complete fallacy.”

Q That's really helpful. Thanks. So how does Deep Blue solve that problem then? Like what are the actual sort of high level mechanics of how it works?

A Yes. So our product writes code, this test code fully autonomously. So very different from LLMs. And in order to do that, you know, we use reinforcement learning to help us find those tests. So the, the basic steps on the first thing we do is analyze the program. So we look at, we see the entire source code of the application. So again, different to LLMs where you've got limited context, you've got before the insertion point, after the insertion point, we've got the entire program. And so we do analysis on that, and that enables us to come up with our first guess, essentially, of our first prediction of what a good test would be. So we take it method by method, so all the functions of the program So enumerate all of those, you understand, uh, exactly, um, uh, the context of, of each method and how you're supposed to call it and what you're supposed to feed into it, all of those things.

AI assessment note: “we use reinforcement learning to help us find those tests”

Q like how things work, let's, um, let's backtrack a bit and talk about what that is. So to make it interesting to a broad audience, uh, yeah. With the, uh, the, you know, intricacies of, of this. So what, what is, uh, what is a unit, I guess, and what is, what is unit tests and why did you all pick that area compared to like other areas of code?

A Yeah. Yes. So the idea with unit tests is you isolate one module of the software and you test everything inside of that in sort of in isolation. So, um, the, you know, unit tests have been around for about 25 years, uh, as a, as a concept. And you can think of it if we, if we make a simple analogy about, um, Production line for a car, right? You, when you're making a car, you check that the brakes work when you put them on the car. You don't wait until the test drive at the end of the production line to find out if the brakes work. And so the same idea in software is called shift left. Now that's a, that's a crude analogy, but it, but it works. It gives you the right idea. You're essentially trying to make it so that you find defects as soon as possible. And so the idea with unit tests is that a developer can run them and they run quickly. And they run them at the time they're writing the code so that failures in unit tests can very quickly be fixed by the developer because they're in the headspace of that code. They've got, you know, they know exactly what they're doing. They've probably been working on this for a while. Everything is ready, you know, there for them to be able to make a change to the code and fix an issue that they find. And that's one fewer defect that then shows up in QA because fixing bugs that are found in QA takes much, much longer. Um, the idea.

AI assessment note: “you isolate one module of the software and you test everything inside of that”

Q for, uh, a broad group of people. Typically the developers would not test and they would just like write code. And then this would go into a separate group of people, uh, who do, uh, would actually test. And, and the idea is that the developer should be testing, you know, where that they were not originally, is that part of the idea was that too much of a caricature?

A Um, yeah, it's, you know, it's a simplification, but yeah, it's, it's pretty much what, uh, what happens. I mean, so when a developer is, uh, working on something, they will test it themselves. The question is how they might do that. They might just test if that piece works. They might build the entire application and, you know, maybe if they're making a change to how the interface works, they go in and make sure that when they do the things that the interface does, the right things happen. Um, the, and so part of the challenge with that is that one, you need to build the entire application and to run it, you might need dependencies that are like a database or something like that. So it's, that can end up being quite slow. So what developers try to do is they don't really want to have to deal with that. So they try and simplify things for themselves and get enough confidence. So it's not like developers are reckless, but Um, there are changes that they're putting in plus the change that other people are putting in and then, um, you know, sometimes, you know, the software doesn't build it and it's the, you know, it works on my machine problem, right?

AI assessment note: “it's a simplification, but yeah, it's pretty much what happens.”

Q What kind of tools, what kind of tools do they use to do that?

A So it depends what kind of software you're building, but let's say if you're building a web application or a mobile application, a lot of it is, you know, clicking on buttons and making sure the right thing happens. So working through the interface is typically how they would do that, and, and, um, in the sort of bad old days, that was all done manually, so people literally sitting in front of screens, clicking on buttons. Now, and obviously everyone wants to automate that, and there's a vast number of products out there to automate UI testing these days, and, you know, web servers, web applications and browsers, and has made that easier to automate, um, and so you can sort of, you can automate the manual process out of this, The fundamental issue though is still that all you know when it, when the software doesn't work is that it doesn't work. You don't know why. So the next thing has to happen is the triage of like, okay, it does the wrong thing. Why is it doing the wrong thing? And sort of working backwards from that problem to what's, what's the root cause that by itself might take days or weeks of work, uh, depending on how complex things are. Uh, and especially in complex systems where you've got other, you've got databases, you've got caches, you've got all these other things happening all at the same time. You get into these problems that are hard to reproduce because t…

AI assessment note: “there's a vast number of products out there to automate UI testing these days”

Q that it's, uh, fully autonomous, like versus again, uh, you know, copilot, uh, which, uh, as the name indicates, has more of that copilot, uh, kind of, uh, Uh, kind of, um, uh, interaction mode, uh, where, where the human may feel like they're more in control. Like, this is, like, autonomous AI. It's like, are people, uh, funded by it, spooked out by it? Like, how do they react?

A Well, we said, well, we offer both models as well. So, you know, we offer an interactive model where you can just point to your code and say, give me a test, and we will just give you tests then and there. Um, and so that's, you know, much more, uh, like a developer is used to. But they might be doing that in the context of, you know, we, previously we wrote 20,000 tests, and so they're in, now they're in test maintenance mode, but they've written some new code, they need tests for it, or they're maintaining, they're maintaining the code, and so that the test drops elite, they need new ones to replace the old ones, and they can do that interactively. So we've made the product so, you know, the most important thing is it works the way the developers are comfortable with working. Um, you can go the full on, fully autonomous route, where the developers Um, don't use the tool interactively at all. They just get the test. They just, they're there in the baseline of the code. They can run them. When the tests are obsolete, uh, they can just delete them, uh, because, uh, we can just regenerate new ones. Um, so that's, uh, so that's a very, you know, for some developers, that's like, that's a little, that's a bridge too far. So it's really what people are comfortable with.

AI assessment note: “for some developers, that's like, that's a little, that's a bridge too far.”

Q And, uh, so now with Deep Blue and reinforcement learning, then, um, how does that, how does that work?

A So the challenge for a lot of organizations, and this is sort of how DifBlue got founded, you know, your question earlier on about that, is that, um, you know, unit testing is a tedious, error-prone stuff to do. It can be quite difficult code to write, and the idea was, because you have to test a single unit, um, you have to figure out how to isolate just that unit, and so for a, you know, database-driven application, we are expecting to have a database on, on the back end, Um, you don't want to have to do that anymore, so you're gonna have to figure out how to, um, essentially emulate the database, and so you're looking at trying to, uh, synthesize test data or synthesize dummy data, pretend, you know, to get the same thing back from the database that you would get, and, and that's called mocking. So that, that process itself is Very tedious. You have to get exactly the right configuration. You have to use a mocking subsystem. You have to learn how to use that, that by just by itself can get very complicated and very slow. And this is why people like developers, um, don't like writing these tests and fundamentally they're not paid to write tests. They're, they're paid to deliver the functionality, the new things in the application that fixes the application, the updates and tests are just there to, as part of the process to help them. But it, What it means, especially in large…

AI assessment note: “So the challenge for a lot of organizations, and this is sort of how DifBlue got”