The Exchanges

Q the last time we saw each other in person was just how quickly, like, the AI, um, ecosystem and research field is evolving and what it means to manage an open source project through that. Can you talk a little bit about what you decide to keep stable and change when you both have, like, big ecosystem of users now and, like, very rapidly changing environment of applications and technology?

A That's been a fun exercise. So I mean, if we go back to the original version of link chain, what it was when it came out was essentially three kind of like high level implementations. Two were based on research papers, and then one was based on Nat Friedman's like Nat bot type of agent web crawler thing. And so there was some high level kind of like abstractions. And then there was a few like integrations. So we had integrations with, I think like open AI, coherent hugging face to start or something like that. And those two layers have, kind of, like, remained. So we have, you know, 700 different integrations. We have a bunch of, kind of, like, higher level chains and agents for, for doing particular things. I think the thing that we've put a lot of emphasis in, um, to your point around, kind of, like, what's remained constant and what's, uh, and what's changed is, like, a lower level, kind of, like, abstraction and runtime for, for joining these things together. One of the things that we pretty quickly saw was that as people wanted to Improve the performance, go from prototype to production. They wanted to customize a lot of these bits, and so we've invested a lot in a lower level kind of like chaining protocol, so lane chain expression language, and then in, in a different protocol lane graph, which is one, something we're really excited about, and that's more aimed at, ah, b…

AI assessment note: “what's remained constant and what's, uh, and what's changed is, like, a lower level”

Q a combination of that and tree search and just, like, trying to be efficient with, like, your sampling at every step has shown, like, a lot of really interesting, uh, effective applications recently, and I think the, like, cognition is one example of, like, a surprisingly amazing agent has, has come out, like, Where else do you think agent, agentic applications will begin to work or that you've already seen?

A I think on the customer support side, that's a pretty obvious use case. I think Sierra, um, you know, has emerged there and is doing, is doing quite well there. Um, I think, yeah, the cognition demo was very impressive. I think they did a lot of things right. I think they really nailed a really interesting UX. Um, and that was maybe one of the things that, that I was most excited about. Um, and then obviously it seems to work very well. And so I don't know exactly what they're doing under the hood. Um, Uh, but, but those type like coding, coding problems in general, we see a lot of people working on. I think there's a really nice feedback loop that you can get by just like executing the code and seeing if it works. Um, and you know, as well as the fact that the people building it are developers and so they can, they can, uh, test it. Um, coding, customer support. There, there's some interesting stuff around like recommend, like recommendation, um, chatbots almost. Um, so I draw a distinction between that and customer support or with customer support, you're maybe trying to explicitly kind of like resolve a ticket or something like that. And the, um, and the recommendation bit is a bit more focused on like a user's preferences and, and what they like. Um, and I think we've seen a few, uh, I think we've seen a few things emerge there. Um, but I'd say customer support and coding a…

AI assessment note: “I think on the customer support side, that's a pretty obvious use case.”

Q any given application, take your prompts and go from, you know, Um, anthropic to mistrawl to OpenAI to something else. Um, in, in reality, it feels like, you know, the way an application, uh, responds is probably going to be sensitive to the fact that these LLMs are actually going to predict differently. Like, what do you think about this? Can you, can you switch? Is that a real pattern?

A It's not as easy as it seems like it should be, and I think the main thing is that the prompts still need to be different, um, for each model. I do think Um, the prompts will probably start to converge in the sense that if you think the models are getting more and more intelligent than like, hopefully these small idiosyncratic sees don't matter as much. Um, and as more and more model providers start supporting the same things, um, then that will make it easier. And what I mean by that is, you know, so many prompts for open AI, which is, you know, the leading and most used one use function calling. Um, and, you know, up until some period ago, like no other models did. And so you just like, couldn't use this prompts at all. Um, but now like Mistral has function calling and, and, and Google has function calling. And so I think they're a little bit more transferable there.

AI assessment note: “It's not as easy as it seems like it should be”

Q getting to performant agents? There's a number of companies that have been started recently that are really focused on sort of the agentic world and pushing that whole thread in certain types of automation forward. What do you view as the big components that you all don't have or that maybe the industry more generally doesn't have that still needs to come into place to help drive those things ahead?

A Yeah, that's a, that's a really good question. I think there's a few things. One, I think like, um, Like figuring out the right UX for a lot of these things is still an open question in my mind. Um, and you know, that's not necessarily something we can help with. I think there's a lot of exploration that applications need to do to figure out how to, you know, communicate what these agents are good at and bad at to end users and expose ways to, um, maybe let them course correct and see what's going on. And so, you know, I think we try to emphasize a lot of this, um, Observability of intermediate steps and even correcting intermediate steps, but, but there's a lot of experimentation around UX that I think needs to happen. Um, another big part I think is, is basically the planning ability of the underlying LLMs. Um, I think that's probably the biggest, I, I think when we see people building agents that work right now, it's often breaking it down into a bunch of smaller components and, and kind of like imparting their domain knowledge about how information should Flow through these components. Um, because I think the elements by themselves still aren't able to, to reason fully about how that should happen. And I think we see a few kind of like, uh, A lot of research is actually around this, I would say, in the academic space. Specifically, I think there are two different types of r…

AI assessment note: “figuring out the right UX... another big part I think is... planning ability”

Q without the actual fine tune. But it's something that feels very, um, awkward on and people talk about it a lot and people talk about doing it a lot. Um, you probably have a great perspective since you see so many different types of customers. Are, are you seeing a lot of fine tuning happening in the wild? And if so, there's specific common applications or use cases for it?

A We see people experimenting with it. I think the only real place where they're doing it is when they've reached like really critical scale, um, which I still don't think is that many applications to, to date. Um, I think there's a lot of difficulties with it. Um, one's like gathering the data set for it. And so I, I think a lot of the things we have in Lang Smith tackle a lot of these issues, but like gathering the data set for it. Um, so like having that data visibility and starting to curate that data set. Um, evaluating the fine tuned model. Um, so like evaluation and testing is a huge pain point there that we're trying to tackle in a few ways. The third is just like, yeah, back to this point of people are still just like experimenting so rapidly. It's much harder to change a fine tuned model than it is to change a prompt or even changed few shot examples. And so I think we're seeing more and more people use few shot examples, um, but not a ton graduating to the fine tuning just because, yeah, I think Uh, much harder to just, like, iterate quickly on.

AI assessment note: “We see people experimenting with it. I think the only real place where they're doing”

Q I guess at the, the high level, similarly, is there anything that you view as like a major prediction or things that'll change over the next year that nobody's really paying as much attention to?

A Memory is a big interest of, of, of ours. Um, and so I hope that we'll have some kind of like breakthroughs there. I think a lot of the specifically around, yeah, learning from interactions, incorporating that back in at a user level, um, in a similar vein, also this, uh, Type of more like system level memory, I think is really interesting and building up, building towards this idea of almost like continual learning. So there's, you know, like, can you learn from your interactions and you can do that in a variety of different ways. This may just be where we sit in the ecosystem, but one exciting, um, and probably under talked about ways is just the idea of, of building up few shot example data sets and really using those. I think it's much faster and cheaper than fine tuning models. Um, it's easier to do than trying to like. Programmatically change the prompt in some way. Like, that's still kind of like a, a, a bit of an art. Um, and so, yeah, continual, towards continual learning with few shot examples is, is maybe one, like, really interesting area that, that we're excited about.

AI assessment note: “Memory is a big interest... one exciting, um, and probably under talked about ways”