The Exchanges

Q So, uh, detailed system prompt, planning tools, sub-agents, file system, is that the list of, like, core components of the modern agent architecture?

A Those are the four that when we launched Deep Agents, and so the story behind launching Deep Agents was we saw, we saw Manus, we saw Cloud Code, we saw Deep Research, they all had these four things, um, and we were like, okay, that's, that's pretty common, let's put it into a Python package and, and, and, um, and make it easy for people to build their own versions of that. So those were the four things at the time. Those are still probably the core things. Some other things that are frequently used. I mean, uh, bash and executing code is, is a big one that's not always used because, uh, sandboxes like Daytona are still new. And so people are still discovering how to run them and how to manage them. And so it's often easier not to do that, but we're seeing more and more want to do that. Um, and so that's where things like sandbox has come in handy. Skills are new primitive that didn't exist when we launched deep agents, but are now very, very, very interesting.

AI assessment note: “Those are still probably the core things.”

Q Since you mentioned sandboxes a second ago, since we are the Daytona Compute conference, Zetona being a leader in sandboxes. Let's talk about the compute layer of agents for, for a minute. So starting at a high level, why do agents need a sandbox?

A Yeah, I think, uh, the main reason in my mind, and you should have Ivan on to definitely correct me, but the main reason that we see so far is to, is to write and run code. Um, so, uh, I would draw a distinction between kind of like file systems and sandbox. Access, as mentioned before, you could have a file system interface that actually does not exist in an actual file system. But if some of those files are code, you might want to run and execute those code, that, that code. Why, why, why is that interesting? Why is that valuable? One, like this code could just be like scripts that are loaded beforehand, but they, but you can parameterize them. You can call them as CLIs or something. And that lets the agent, it's a different form of tool calling, um, that can often be easier. Two, the agent can write its own code and then run it. And in particular, like this last one is, is Why you need sandboxes. Anytime you want the agent to kind of like run untrusted code or do arbitrary things, you don't want that kind of like happening on a, on a shared server on your, or even on your like, uh, local computer. I think you see this a little bit with the open claw stuff, right? Like open claw, um, you know, uh, it, it does a bunch of things under the hood, including kind of like writing and running code. That's why people are buying Mac minis as a, you know, um, primitive way of sandboxing…

AI assessment note: “the main reason that we see so far is to, is to write and run code.”

Q What kind of agents are we talking about? Are we talking about coding agents? I think you said somewhere that every agent should be a coding agent.

A So we see a divergence between, like, two different types of agents out there. One of them is, like, conversational agents. So these would be, like, the customer support, customer experience, chatbots. These have, these, like, require really low latency. Voice is oftentimes the medium that they interact with, and that's one style of agents that are mostly, like, conversational. They don't do a ton of tool calling. They'll maybe do, like, one or two, because they can't do too many or will take too long. But then we see this other style of agents, which Sequoia came up with this name, long horizon agents, and I, and I really like that they can operate over long horizons. They can do some planning. They can maintain coherence. And yes, a lot of them end up looking like coding agents. And I think there's probably like, there's a few reasons for that, but one code is really useful. You can do, you can use code to do a bunch of different things. You can use it to parse text files. You can use it to do things programmatically. Like you want to loop over a hundred different files rather than doing a hundred different tool calls. You can write a script that does that. So code is like really generally useful. But then also the models are trained on code. And so all the big model labs have been RL and code and bash and, and, and, and editing files into those models. And so that is the stu…

AI assessment note: “we see a divergence between, like, two different types of agents out there.”

Q As you describe all of this, um, I'm trying to figure out what the concept of memory means, because it seems like there's memory in the file system, there's memory in the sub-agents. Is memory in other places as well? What is memory for agents?

A Memory is super important. I mean, I think a lot of what we've been talking about so far, I would describe as like short-term memory, which is really like within a particular thread or conversation. So even when you summarize, that's still within a particular kind of like thread. The more interesting type of memory I think is long-term memory. And so what long-term memory is, there's three different types of long-term memory. Uh, one is like semantic memory. And so that's basically, you can think of like RAG for that. So there's a lot of facts that Somehow get put into this kind of like semantic store that could be through conversation. So I talked to you, I learned things like anthropomorphizing better here, but like I talked to you, I learned things, I store them in some place and I can go back and say, oh yeah, Matt's, you know, favorite drink is whatever he's drinking at the moment or something like that. And so that's like a semantic fact that I can store that you can think of it. Yeah. Just retrieval rag, um, episodic. And we know how to do that. We know how to do rag and stuff like that. What the interesting part there is how do those things Get into memory. How did those get extracted? That's a little bit more, you know, that, that that's, that's where that's not really figured out. And there's some interesting thinking to be done there. Um, episodic is basically previo…

AI assessment note: “there's three different types of long-term memory. Uh, one is like semantic memory.”

Q Is there another part of the ecosystem that is stable enough that's worth investing into obviously as I'm Listening to you speak, it's such a dynamic field. What about MCP, for example? Has, has everybody normalized on MCP being the standard?

A Yeah, MCP is fine. I mean, it's a way, it's a way to expose APIs in a standard format. Um, it's great. It has, it has a bunch of other kind of like features like elicitation and things like that, that are not supported by nearly as many kind of like clients. I think the core part of like, how do you expose, uh, how do you expose APIs in a standard way is, is definitely useful. I mean, I think the, um, I think the stable stuff is probably stuff that's a little bit more, um, lower level. So we do a bunch with observability. I think no matter what these agents look like, you're going to want to know what's going on inside of them. Um, same with evals, no matter what they look like, you're going to want to measure them in some way. Um, sandboxes, I actually think are a really good example of this. Like they're pretty low level infrastructure piece. You know, if, if agents never write any code, then okay, maybe they're not useful, but I think it's trending where Basically all agents will write code, so that's a very interesting piece, I think. Um, those are, like, the, uh, I think, like, so I think, like, pretty clearly agents will be long-running and stateful, and so I think we have a deployments product. I think a lot of the, uh, I think deployments products that let you build long-running stateful things will be kind of, like, interesting no matter what. And that's kind of how we…

AI assessment note: “Yeah, MCP is fine. I mean, it's a way to expose APIs”

Q And do you think, uh, conversational agents become coding agents as well as they go deeper into the stack?

A This is a really good question. I mean, I, I, we talk a bunch about this internally because we're debating whether we should build like a different type of agent harness for these types of agents. I think, um, I think there will kind of be a convergence when there are agents that can reliably like kick off and manage other long horizon agents. So one of the things that we're seeing in coding is that people want this experience of being able to kick off a bunch of other, Like do a bunch of work, kick off a bunch of agents, but, but keep on chatting with like the main agent. And that's very similar to like a conversational agent in some sense, right? Like it's, it, you've got that, like, you've got that like constant kind of like back and forth, um, latency TPD, but then, you know, the, these voice agents, I think will obviously want to do more and more like long running things in the future. And I think the way that you'd do that is you'd, you'd basically have two agents, one that runs in the background and is kicked off by this other kind of like conversational agent. So it could all kind of like converge into this, into the single harness that just supports basically long running async background agents as a tool.

AI assessment note: “I think there will kind of be a convergence when there are agents”

Q So seen from LinkChain as a company, uh, LinkChain's perspective, sandboxes are something to recap that, that you call, uh, what's your surface area of, of contact with the sandbox?

A So I think there's two interesting ways that agents can use sandboxes. One, you can basically spin up the sandbox and then install the agent there. And have the agent running inside the sandbox. Another way to use sandbox is you can actually have the agent running outside and then have it call the sandbox like as a tool. Um, and in, in practice, we see people doing about fifty-fifty between each of these. I wrote a Twitter article on this and people from both sides yelled at me and were like, how can you even say there's another option? It clearly has to be X or it clearly has to be Y. So I do think it's a little bit up in the air. One thing that I'd maybe say is like, I think a lot of these agents Um, a lot of these agent harnesses are coming from the coding agent world. And if you look at like something like cloud code, it's very much built to be run on kind of like your, your local machine or your local kind of like system. And so in that, so people who are coming from the world of like, oh, I see cloud code. I'm going to take cloud code or cloud agent SDK and run it. They almost always spin up a sandbox and then install cloud code in there because that's the, that's the way it's meant to be run for people who are coming at it more fresh or holistically. And they're like, Hey, you know, I've got this agent. I want to give it coding ability. That's where we see people spinnin…

AI assessment note: “there's two interesting ways that agents can use sandboxes”

Q Great. And to make this, uh, broadly, uh, accessible and interesting, uh, uh, for, you know, large group of people, what's a harness in plain English?

A Is, is how the model kind of like interacts with its environment is what I would say. So it's the set of tools that it has. Um, it's the, and some of these tools can be really specific and I actually wouldn't count those as part of the harness, but some of these tools can interact with a more general environment. So if we think about coding agents, I would say like the file editing tools it has are like part of the harness. I would say the, the ability to run a code is part of the harness. If, if you take a harness and give it a particular tool for interacting with Slack, I would argue that's you kind of like customizing and building on top of the harness. And that's how we think most agents should be built. Like we think most agents should be built by taking a harness Um, and, and giving it some instructions and giving it some, some kind of like tools and those tools could be specific tools like a slack tool, or they could be kind of like configurations of tools that are built into the harness. So what I mean by that is most harnesses today have sub agents built in. They have, uh, skills built in. And so you could configure them with particular skills, but the fact that those like skill abstractions and sub agent abstractions exist, like I would argue that's part of the harness. Other things that the harness does is like take advantage of like prompt caching. Um, it does, uh, …

AI assessment note: “Is, is how the model kind of like interacts with its environment is what I would say.”

Q Do you want to explain what skills are?

A Yeah, skills are great. So they're basically like a bunch of files. There's usually one kind of like skill.md file, which is a big markdown file that contains instructions on how to do something. And there could be other things in a skill as well. There could be other scripts that it could run, but it's basically these instructions for how to do particular things. And rather than being loaded into the system prompt, they are just like referenced in the system prompt. So you'll tell the agent, Hey, you have access. To like this code writing skill and you have access to this documentation skill. And then if it decides that it needs to use those skills, it will, it will just go basically read those files on demand. People call that kind of like progressive disclosure. You tell the album only what it needs to know when, when it needs to know, it's another way of letting it manage its own context window as well. So that's a key part that we support in deep agents and most harnesses support. Um, I mean, other interesting things that we're thinking a bunch about, like async sub agents are really interesting. Um, I mentioned this earlier, but like, I, I think this is something that most like harnesses don't do that well. I think technically cloud code has support in it, but I don't even know when it triggers it or, and it's hard to like observe them and manage them, but I think this wi…

AI assessment note: “So they're basically like a bunch of files. There's usually one kind of like skill.md file”

Q Where do you think this all goes? Um, as each agent, uh, accumulates more memory, more context, do you end up with one agent that can do it all or like a fleet of thousands of agents and sub agents? That get orchestrated?

A It's a good question. I mean, I think, I do think that like memory defines an agent. I think the interesting thing is that you can, you can take the memory that defines an agent, like the system prompt and the skills it has, and you can just expose that as kind of like a skill to one mega agent. Um, so like we get asked a bunch about like a common thing that we get asked about is people are building these agents in enterprises. They have like 20 different organizations. They know that they want each organization to basically, you know, build something agentic, but they, they want there to be the kind of like one interface that controls all 20. So a very common thing is like, how do we, how do we do this? And the right answer to that changes a bunch. And, and it's actually unclear what the right answer is right now. Like, is it one big agent and then, and then it has like skills for each of the 20 divisions or departments? Is it 20 kind of like sub-agents? Um, is it 20 like completely custom kind of like workflows and stuff like that? The, the, the, the answer changes a bunch. The things that the, the things that I absolutely believe are that the, the most important things for all of those divisions to build up are like the instructions and the tools themselves. And then whether those get bundled as a skill or bundled as a sub agent, or they even build their own kind of like age…

AI assessment note: “it's actually unclear what the right answer is right now.”

Q The topic of evaluations is fascinating. It seems that there is a trend now with co-work where the end user has the ability to evaluate and provide feedback to the system. How do you think about how to build the proper harness for this so that companies can build agents that continuously improve on a per user basis?

A Yeah, there's, there's some really interesting tie-ins between evaluation and memory and prompt optimization as well. Those are all kind of related because all of them basically involve the agent doing something, some reward function for what the agent does, and then updating some kind of, and then optionally updating some parameters. Um, so if you're doing kind of like what we would call offline evals, like, you know, you, you've got a, uh, you've got an agent, you're about to ship to production. You might want to do offline evals. You take your agent, you run it over some dataset. You then, ah, you take all those examples, you score them with some functions, and then you check to make sure there's no regressions, or you manually change the agent. For, like, for memory, um, which is what, like, co-work might do when it remembers things, You, as a user, use the agent on one thing, you tell the agent it did something bad, and then the agent updates its instructions. So that doesn't happen again. And then same with prompt optimization, prompt optimization. You do the same thing as online evals. You run it over a bunch of data points. You then run your evaluators, but then you take all that feedback that you get and you have the agent update the prompt according to all of that. So I think it's all kind of related and right. It's all like similar concepts, but they are pretty like …

AI assessment note: “you tell the agent it did something bad, and then the agent updates its instructions.”

Q And how does the system know to create a sub-agent?

A It's all in the prompt. It's all in the prompt. Yeah. That's the beauty of these types of agent harnesses. You know, like earlier when we were doing things with line graph, people would be like, okay, how do I add like a step to make sure that the agent does this before X? Or how do I, how do I, you know, enforce that the, the, for better or worse, and this is why, uh, this is why line graph still has a place. I'll get to that later, but like for better or worse, the way that you Get these things to do anything is you just tell them to do it. Um, and, and, and that's great cause it's like flexible, but that is also not like a hundred percent reliable. And so we actually see still pretty good adoption and pickup of land graph in heavily regulated industries where you want like a ton of control and precision, um, and reliability, because as good as these kind of like coding agents are, they are pretty unpredictable in terms of what they do. And, and, and there's no guarantees on anything. It's why they're so. Enticing because you just tell them to do things and they do things, but there's no guarantee. And so that's a downside as well.

AI assessment note: “It's all in the prompt. It's all in the prompt.”