The Exchanges

Q I saw an answer, something called tools. Did we cover that already? Is that, is that, is that, so we talked about product engineering, we talked about fine tuning.

A So we haven't actually covered tools yet. So, you know, LLMs by default are text in text out. And so they're limited in what they can do in terms of taking action in the world. And the idea of tools. Or what OpenAI originally called function calling and we sort of called it tools. And now I think we, we won the war on the naming because everyone, they renamed it to tools now as well. But essentially what this is, is that if an LLM wants to take an action in the world, you essentially allow it to do that by exposing a set of APIs. So a set of different programs that the LLM can use. And you say to the model, Hey, these are the tools that are available to you. So for example, web browsing might be a tool that's available to the model. And the model then can output a request to that API, which is just a JSON string. So it's just another piece of text. And that JSON string will specify which tool it wants to use, what question it wants to ask of that tool or how it wants to use that tool. That output is then taken, used to run the tool itself. So maybe you'll go and do a web search. The result of the web search is then passed back to the model and the model then uses that output to make another decision. And so suddenly you go from a system that's only text in text out to something that can be action taking and that can actually take advantage of external APIs in the world or do in…

AI assessment note: “So we haven't actually covered tools yet.”

Q Okay, great. So what would you say are the specific challenges when evaluating and monitoring LLMs? Obviously the category of evaluation and monitoring in the software world has yielded huge companies like Datadog in particular, but that's one world. Like how is the world of LLM different? What are the specific challenges and opportunities?

A Yeah. So machine learning differs from traditional software in a couple of key ways and then generative AI as a kind of subset of machine learning differs even further. So the first transition you go to when you go from kind of traditional code to machine learning is that it's no longer deterministic, right? We're used to, for software engineers, writing a program, you run it, you get the same results each time. You can write a deterministic test and people are doing performance monitoring with something like Datadog, but they're not expecting that when they run the code each time, they're going to get different outputs. Once you move to the world of machine learning, now it's stochastic. And not only that, but you're now specifying what the program does via a data set and a training process rather than, you know, deterministically in code. And so evaluation and machine learning focuses on accuracy metrics and things like that. And when we go to LLMs and generative AI, we go one step further where the use cases that people are applying these to are very general and very subjective often. To pick a couple of examples, you know, if you're helping someone draft a sales email, Um, or you're writing marketing copy. There isn't any longer a kind of ground truth answer that you can compare against for the model to know whether it's correct or not. Even if you're doing a question answe…

AI assessment note: “There isn't any longer a kind of ground truth answer that you can compare against”

Q So how are you solving the problem? Maybe taking those three in turn, what human look do to address the issue.

A Yeah, so we provide the ability to get evaluation data, both from human feedback in various ways and also in an automated way. So I maybe take the human feedback version first, and this is actually the first version of human loops product started doing this, which is that because it's very subjective, there's some sense in which the only real ground truth is your customer satisfaction or opinion of the experience they get out of the product. And so we make it really easy for people to instrument their applications with the ability to capture feedback. So the simplest version of this is things like thumbs up, thumbs down that you might have seen in many applications. You see it in chat GPT, but people also tend to collect implicit signals of user satisfaction. So the actions that they take after interacting with a particular LLM app. And also if people are able to edit any generated texts or generated content, then also capturing those edits can be a very useful signal of how well things are working. And then in the human loop app, we triangulate those sources of feedback. Back against, okay, what model created it? What inputs created it? And we give product teams the ability to dig into that data, analyze it, understand what's working well and isn't and why, and then critically to also in the same application, take actions to change things, to make them better. So to edit promp…

AI assessment note: “we provide the ability to get evaluation data, both from human feedback in various ways”

Q And hopefully not an unfair question because Datadog has built, we know a multi-billion dollar business, but let's assume human loop finds regression or, you know, different results from the same prompt. Then what can you do as an enterprise? Obviously knowing that this is an issue is essential, but is there a way you can fix that or you should just be aware?

A No, it's, I think this is one of the strongest arguments for why a new set of tools is needed and why using, you know, existing platforms for monitoring and observability is less productive. And it's because with generative AI, the speed with which you can make interventions is extremely high. And having that in one combined platform is actually one of the powers of this. So you're absolutely right. So within human loop I mentioned, we have this kind of interactive environment, both for, for prompt engineering And we also have the ability for people to fine tune models, which is where you do a little bit of extra training on a new data set. And so what will often happen is people will find a bug, you know, they'll basically like be exploring the, the log data within human loop. They'll find an issue. They'll reopen those data points back into that interactive environment where they can now run what if style analysis. So they can change the prompt or they can change the information retrieval system a little bit. And see what the impact was. If they're able to then fix that issue, they then run a regression test. They say, okay, is this new prompt still performing well on what worked before? And if the answer is yes, they can actually promote it straight to development or production from within that system. And so actually a product person or a domain expert who's able to go in a…

AI assessment note: “They'll reopen those data points back into that interactive environment where they can now run”

Q own evolution. And as a result, it's a little, uh, a little blurry who does what. So any thought there would be very helpful to in particular, Frameworks like lane chain. Where, where does that fit? Are you a competitor? Are you a partner? And then a JGPD enterprise or what OpenDI does in terms of like getting further into the enterprise. Same thing is that a competitor, friend, foe.

A Yeah. So I think at base you have the foundation model providers, right? So here you would have OpenAI, Cohere, Anthropic, Mistral, whoever else it might be, the open source models, Llama. And then at the other end, you know, the other end of this spectrum maybe is the applications, which are the end user facing applications. And human loop is kind of a layer that sits in the middle. So we're model agnostic, we have close partnerships and we'd definitely be friends with all of the foundation model providers. We're keen to basically help their customers get to value. LangSmith and LangChain. LangChain is like a orchestration library. So this is basically just a set of utility tools for people who are writing the code around an, an AI application. And it has a whole bunch of helper functions built in that help them get started more quickly. So that wouldn't be sort of directly competitive. Their, their LangSmith product probably has a little bit of overlap with us, but it is not, you know, it has some overlap that is fundamentally, I think focused more around monitoring chains and agents. And then where I think we are focused is for enterprises who are building LM aspects where typically there's a lot more collaboration required. This becomes a team sport. And also where the need to have guardrails and evaluation starts to become much more significant because they're operating at…

AI assessment note: “we'd definitely be friends with all of the foundation model providers”

Q know, some of the key debates in the industry right now. So certainly as we are recording this, there seems to be a tweet every second on open source, the closed source models and, you know, people that feel very strongly about preserving a very Free open ecosystem and others that are more concerned about security risk. Any, any, any thoughts on that debate? Where do you land? Just curious.

A My natural inclination is always to be in favor of open source, right? Just as a default knee jerk reaction, we've got so much benefit from open source in general. The software world is built on top of it that, that I always start from a position of optimism about open source. I understand though, some of the reasons why people have Safety concerns around larger models. There is an opportunity for misuse. And I've seen people like Jan LeCun say, oh, but we already have search engines. So like, you know, people can look up with a search engine, how to build a bomb or how to, you know, build a bio weapon. Like why do LLMs make it worse? But I think that really does downplay like how much better they are at synthesizing information and explaining steps to you. And there is a dramatic reduction in how hard it is to do certain forms of misuse. And that's before we get to the more safe, you know, the safety concerns about AI that actually is misaligned. But to me, the question is like, it's very difficult to know when to, like how to solve this or when to put restrictions in place. People, when GPT-II came out, they didn't release GPT-II for fear of misuse concerns. And then GPT-III, and now we're on GPT-IV. It would have been really sad for the world, I think, if at the point of GPT-II, we had decided, hey, you know what? This is too dangerous. No one can have access. Cause we would…

AI assessment note: “My natural inclination is always to be in favor of open source”

Q clarifying that it's not what you Cover, but just out of curiosity, have you observed any kind of best practice around what you've seen customers do on prompts, like people in whatever marketing or finance or HR, like any way to guide them on how to best use those products in a way that just minimizes the issue, or is that still very new territory and people are finding out?

A Yeah, we, it's, it's funny. We do have a couple of customers who use human loop in that way, because I think that actually the market is missing an appropriate solution. And so they kind of Hack what is not built for that for a slightly different use case, but we have some customers where actually they have maybe a thousand users who are on human loop and where they're saving common prompts and workflows on a per team basis. So they say, okay, here's how you, you know, this is the way that you solve X task with a, with an LLM and there's a predefined prompt and they go and they open it up, they load it, they dump in their data, they run that process. Honestly, I think that it's something that companies are still figuring out and there isn't a very good solution on the market yet. It's something that I would hope ChatGPT Enterprise just solves at some point by, you know, things like their custom GPTs, I think are a step in this direction of allowing people to save workflows and prompts and share them with the team. I haven't seen other good solutions yet. I'm reasonably confident that it's not well solved because of the fact that I see people kind of using an abusing human loop in this way.

AI assessment note: “they're saving common prompts and workflows on a per team basis”