The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 22 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Albrecht: Optimizing for competitive coding benchmarks does not create useful programmers
“Like, we do a lot of code generation, but we don't really do a lot on, like, code competition problems for the very, very hard ones, so that you can go very far down that route and make something like really good at those problems, but not actually that useful…”
Albrecht: Training on AWS prevents diagnosing low-level hardware errors
“And if we're just using, you know, AWS or some other cloud provider, These errors are still going to be there, and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong.”
Kanjun Qiu: Most autonomous AI agents do not work well today
“So to your question of, like, what agents work well and what doesn't work well, like, most of the agents don't work well, and we're slowly making them work better by improving the underlying model and improving these.”
Qiu: Reasoning is the single biggest blocker for AI agents
“Reasoning is actually, we believe the biggest blocker to agents or systems that can do these larger goals.”
Qiu: Imbue generates specific reasoning data rather than relying on web data
“So I think internally, yeah, we have a lot of thoughts on what reasoning is, and we generate a lot more specific data. We're not just like, oh, it'll figure out reasoning from this black box or like, it'll figure out reasoning from the data that, that exists.”
Albrecht: Dynamic data mix schedules during LLM training yield negligible gains
“We did some experiments and we've actually talked to a bunch of researchers who were doing work here as well and looking at kind of their experiments on this. And we were originally pretty hopeful because it sounds like something that should work and make sens…”
Albrecht: Imbue avoids Kubernetes to keep cluster infrastructure simple to debug
“Less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes, for example, I would just directly launch these things and it's just been much easier to debug this way.”
Albrecht: Vision is not essential for most coding and reasoning agent tasks
“And actually we found that for most of the kind of like code writing and reasoning problems that we care about, the visual part isn't really a huge important part of it.”
Albrecht: Code execution expands agent capabilities far beyond hard-coded tool calling
“Instead of worrying about like weird hard coded agents using tools, Like let's just make them able to actually write code robustly and make that code work and be able to debug that code, know if that code is safe to run, like get really good at the like code w…”
Albrecht: Imbue's GPU cluster failure rate is well below industry 3% benchmark
“The number that we've heard from other people is like they're having about three percent. I don't think we're experiencing failure rates that are that high. I think ours is actually quite a bit lower than that, probably because we've taken the time to like dig…”
Albrecht: Imbue reproduced 500-1,000 examples per dataset to stop eval contamination
“Let's just reproduce, you know, 500 to a thousand examples for every single one of these data sets ourselves and just make sure that this data is definitely not in the, you know, the training set. So we did that and then we're able to like now be confident abo…”
Albrecht: Coding agents communicating uncertainty are far more useful than slightly more accurate ones
“I would much rather have a coding agent that will give me back a thing. And you know, it's actually the code doesn't work like 10% less of the time than some other model, but it will tell me a hundred percent of the time. When it got like when it's not sure, l…”
Imbue built an AI agent to analyze 20,000 Commerce Department policy proposals
“We built an agent that helped us analyze the, like, 20,000 pages of policy proposals submitted to the Department of Commerce request for AI policy proposals, and we, like, looked at what were the problems people brought up, and what were the solutions they pre…”
Kanjun Qiu: Agent development will evolve beyond bare metal to higher abstractions
“And I think it's basically a similar route here where we're like in the like bare metal phase of agent building, and we will eventually get to something with much nicer abstractions.”
Albrecht: Imbue Manages Infrastructure with Three to Six Engineers
“Like our infrastructure team is like
You know, it fluctuates from week to week, depending on like how many things are on fire and how much we need to build.
But it's like between like three and six people, like it's small.
It's not like some huge team of like …”
Albrecht: 4K GPU clusters require 3-tier networking versus standard 1K 2-tier setups
“The normal, the like vanilla setup or, you know, these large clusters as vanilla as it can be is what's normally like a 127 node cluster. So closer to like 10, 24 GPUs instead of 4000. Here we have a larger cluster. As you start to get into the larger clusters…”
Albrecht: Imbue is releasing a new reasoning benchmark and 11 cleaned evaluations
“We're releasing a whole bunch of different data there, a new benchmark about code, reasoning, understanding, as well as our own private versions of 11 different open source benchmarks. So things like PoolQ or ANLI, where we've gone through and kind of cleaned …”
Imbue is releasing approximately 450,000 human evaluation judgments
“A final thing that we're releasing there is around 450,000 human judgments about ambiguity and question quality, which we used In the process of cleaning these evaluations”
Albrecht: CARBS models compute cost per sample for hyperparameter search
“CARBS is, it's maybe a backronym, but it's for Cost Aware Pareto Region Bayesian Search... The point is that it's a cost aware hyperparameter tuner. So most hyperparameter tuners you kind of say, okay, here's this objective function. I want you to make this nu…”
Albrecht: Imbue will not release model weights, but will open-source training tools
“We're not releasing the model. We're not releasing the weights, but we are releasing a bunch of different things that should make it easier for other people to make their own models.”
Albrecht: 4,000-GPU Three-Tier Cluster Requires 12,000 Cables and 24,000 Plugs
“Like to bring up this cluster you know, with 4000 GPUs and three tier networking, networking architecture, you have 12,000 cables.
So that's 24,000 things that need to be plugged in.”
Imbue's CARBS hyperparameter optimizer originated from plasma physics concepts
“CARBS, our hyperparameter optimizer, came from Abe trying to automate his own research process doing hyperparameter optimization, and he actually pulled some ideas from plasma physics, he's a plasma physicist, to make the local search work.”