Imbue

“So to your question of, like, what agents work well and what doesn't work well, like, most of the agents don't work well, and we're slowly making them work better by improving the underlying model and improving these.”

“We built an agent that helped us analyze the, like, 20,000 pages of policy proposals submitted to the Department of Commerce request for AI policy proposals, and we, like, looked at what were the problems people brought up, and what were the solutions they pre…”

“CARBS, our hyperparameter optimizer, came from Abe trying to automate his own research process doing hyperparameter optimization, and he actually pulled some ideas from plasma physics, he's a plasma physicist, to make the local search work.”

“Reasoning is actually, we believe the biggest blocker to agents or systems that can do these larger goals.”

“So I think internally, yeah, we have a lot of thoughts on what reasoning is, and we generate a lot more specific data. We're not just like, oh, it'll figure out reasoning from this black box or like, it'll figure out reasoning from the data that, that exists.”

“And I think it's basically a similar route here where we're like in the like bare metal phase of agent building, and we will eventually get to something with much nicer abstractions.”

“The number that we've heard from other people is like they're having about three percent. I don't think we're experiencing failure rates that are that high. I think ours is actually quite a bit lower than that, probably because we've taken the time to like dig…”

“Like, we do a lot of code generation, but we don't really do a lot on, like, code competition problems for the very, very hard ones, so that you can go very far down that route and make something like really good at those problems, but not actually that useful…”

“We did some experiments and we've actually talked to a bunch of researchers who were doing work here as well and looking at kind of their experiments on this. And we were originally pretty hopeful because it sounds like something that should work and make sens…”

“Like our infrastructure team is like You know, it fluctuates from week to week, depending on like how many things are on fire and how much we need to build. But it's like between like three and six people, like it's small. It's not like some huge team of like …”

“Less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes, for example, I would just directly launch these things and it's just been much easier to debug this way.”

“The normal, the like vanilla setup or, you know, these large clusters as vanilla as it can be is what's normally like a 127 node cluster. So closer to like 10, 24 GPUs instead of 4000. Here we have a larger cluster. As you start to get into the larger clusters…”

“And actually we found that for most of the kind of like code writing and reasoning problems that we care about, the visual part isn't really a huge important part of it.”

“We're releasing a whole bunch of different data there, a new benchmark about code, reasoning, understanding, as well as our own private versions of 11 different open source benchmarks. So things like PoolQ or ANLI, where we've gone through and kind of cleaned …”

“A final thing that we're releasing there is around 450,000 human judgments about ambiguity and question quality, which we used In the process of cleaning these evaluations”

“Let's just reproduce, you know, 500 to a thousand examples for every single one of these data sets ourselves and just make sure that this data is definitely not in the, you know, the training set. So we did that and then we're able to like now be confident abo…”

“CARBS is, it's maybe a backronym, but it's for Cost Aware Pareto Region Bayesian Search... The point is that it's a cost aware hyperparameter tuner. So most hyperparameter tuners you kind of say, okay, here's this objective function. I want you to make this nu…”

“And if we're just using, you know, AWS or some other cloud provider, These errors are still going to be there, and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong.”

“I would much rather have a coding agent that will give me back a thing. And you know, it's actually the code doesn't work like 10% less of the time than some other model, but it will tell me a hundred percent of the time. When it got like when it's not sure, l…”

“We're not releasing the model. We're not releasing the weights, but we are releasing a bunch of different things that should make it easier for other people to make their own models.”

“Like to bring up this cluster you know, with 4000 GPUs and three tier networking, networking architecture, you have 12,000 cables. So that's 24,000 things that need to be plugged in.”

“Instead of worrying about like weird hard coded agents using tools, Like let's just make them able to actually write code robustly and make that code work and be able to debug that code, know if that code is safe to run, like get really good at the like code w…”