Josh Albrecht
Co-founder & CTO, Imbue · 1 appearance on the record.
computed by AI from the episodes · how this works → · full disclaimer →
founderexecutiveengineerscientistinvestor@joshalbrecht ↗LinkedIn ↗joshalbrecht.com ↗
Josh Albrecht is the co-founder and CTO of Imbue, an AI research lab developing reasoning-focused AI agents. He previously co-founded AI recruiting startup Sourceress, worked as an infrastructure engineer at Addepar, and is a general partner at Outset Capital.
22statements → 6claims → 4claims resolved → 100%fully supported → 4/5average certainty → 2.09/5average debate potential → 8said about them ↓
4 supported 0 partly supported 0 contradicted 2 not checkable as stated how the 6 claims stand · each chip opens the sources
6 assertions · 1 opinion · 9 insights · 6 disclosures · every statement was checked. The predictions and assertions are the 6 claims: statements the public record can support or contradict. 4 are resolved, and 2 name no date, number or outcome precise enough to check. Everything else (opinions, insights, what ifs, disclosures) can never be settled by the record, so it carries no assessment.
The record, in short
What the tape says about how Josh argues and how the claims held up. Everything they said, and everything said about them, is in the tabs below.
Their most notable supported claim
Albrecht: Benchmark performance differences vanish once ambiguous questions are cleaned
“The main takeaway from any of the, like, actual performance is like, once you fix these ambiguous examples, a lot of these benchmarks are really saturated. Like, I think it's important to look at like, you know, like when you're talking about performance on NL…”
Everything Josh Albrecht said on Latent Space that made the record, most notable first. Filter by type, assessment or year in the ledger →
Albrecht: LLM emergence is an artifact of non-linear evaluation metrics
“This emergent behavior that you're seeing, Is not really emergent behavior, but is really a function of the evaluation metrics that we're using.”
Albrecht: Training on AWS prevents diagnosing low-level hardware errors
“And if we're just using, you know, AWS or some other cloud provider, These errors are still going to be there, and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong.”
Albrecht: Benchmark performance differences vanish once ambiguous questions are cleaned
“The main takeaway from any of the, like, actual performance is like, once you fix these ambiguous examples, a lot of these benchmarks are really saturated. Like, I think it's important to look at like, you know, like when you're talking about performance on NL…”
Albrecht: AgentBench paper's appendix examples are actually incorrect solutions
“Like we were looking at the agent bench paper, I think just last week for our paper club. And one of the things that we noticed is that actually like both of the examples in the appendix that are given as like traces where it got it right. This is actually not…”
Albrecht: Optimizing for competitive coding benchmarks does not create useful programmers
“Like, we do a lot of code generation, but we don't really do a lot on, like, code competition problems for the very, very hard ones, so that you can go very far down that route and make something like really good at those problems, but not actually that useful…”
Albrecht: Imbue avoids Kubernetes to keep cluster infrastructure simple to debug
“Less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes, for example, I would just directly launch these things and it's just been much easier to debug this way.”
Albrecht: Vision is not essential for most coding and reasoning agent tasks
“And actually we found that for most of the kind of like code writing and reasoning problems that we care about, the visual part isn't really a huge important part of it.”
Albrecht: Dynamic data mix schedules during LLM training yield negligible gains
“We did some experiments and we've actually talked to a bunch of researchers who were doing work here as well and looking at kind of their experiments on this. And we were originally pretty hopeful because it sounds like something that should work and make sens…”
Albrecht: Code execution expands agent capabilities far beyond hard-coded tool calling
“Instead of worrying about like weird hard coded agents using tools, Like let's just make them able to actually write code robustly and make that code work and be able to debug that code, know if that code is safe to run, like get really good at the like code w…”
Albrecht: Messy real-world data limits knowledge graphs to niche problems
“But I think in the real world, it gets a lot messier than like knowledge graph style of things where it's like, well, is there a relationship between these two nodes? Like, ah, I don't know. Like is, are these two separate nodes? Like those kinds of messy bord…”
Albrecht: Imbue's GPU cluster failure rate is well below industry 3% benchmark
“The number that we've heard from other people is like they're having about three percent. I don't think we're experiencing failure rates that are that high. I think ours is actually quite a bit lower than that, probably because we've taken the time to like dig…”
Albrecht: Cost-aware tuning reveals scaling laws for all hyperparameters
“So by doing that, we can see the scaling laws or not just, you know, the scaling laws from like the, you know, chinchilla paper, the scaling laws for all parameters. We can see how does the number of layers change with this? How does the You know, the learning…”
Albrecht: Imbue reproduced 500-1,000 examples per dataset to stop eval contamination
“Let's just reproduce, you know, 500 to a thousand examples for every single one of these data sets ourselves and just make sure that this data is definitely not in the, you know, the training set. So we did that and then we're able to like now be confident abo…”
Albrecht: Coding agents communicating uncertainty are far more useful than slightly more accurate ones
“I would much rather have a coding agent that will give me back a thing. And you know, it's actually the code doesn't work like 10% less of the time than some other model, but it will tell me a hundred percent of the time. When it got like when it's not sure, l…”
Albrecht: Imbue will not release model weights, but will open-source training tools
“We're not releasing the model. We're not releasing the weights, but we are releasing a bunch of different things that should make it easier for other people to make their own models.”
Albrecht: Imbue is releasing a new reasoning benchmark and 11 cleaned evaluations
“We're releasing a whole bunch of different data there, a new benchmark about code, reasoning, understanding, as well as our own private versions of 11 different open source benchmarks. So things like PoolQ or ANLI, where we've gone through and kind of cleaned …”
Imbue is releasing approximately 450,000 human evaluation judgments
“A final thing that we're releasing there is around 450,000 human judgments about ambiguity and question quality, which we used In the process of cleaning these evaluations”
Albrecht: 4K GPU clusters require 3-tier networking versus standard 1K 2-tier setups
“The normal, the like vanilla setup or, you know, these large clusters as vanilla as it can be is what's normally like a 127 node cluster. So closer to like 10, 24 GPUs instead of 4000. Here we have a larger cluster. As you start to get into the larger clusters…”
Albrecht: Imbue Manages Infrastructure with Three to Six Engineers
“Like our infrastructure team is like
You know, it fluctuates from week to week, depending on like how many things are on fire and how much we need to build.
But it's like between like three and six people, like it's small.
It's not like some huge team of like …”
Albrecht: 4,000-GPU Three-Tier Cluster Requires 12,000 Cables and 24,000 Plugs
“Like to bring up this cluster you know, with 4000 GPUs and three tier networking, networking architecture, you have 12,000 cables.
So that's 24,000 things that need to be plugged in.”
Albrecht: Unsynchronized garbage collection across distributed nodes steadily degrades cluster MFU
“Because you have hundreds of machines, they're doing garbage collection at slightly different times, and then they get slightly further apart, and slightly more and more jittered, until eventually they're all happening kind of at random times, and just, like, …”
Albrecht: CARBS models compute cost per sample for hyperparameter search
“CARBS is, it's maybe a backronym, but it's for Cost Aware Pareto Region Bayesian Search... The point is that it's a cost aware hyperparameter tuner. So most hyperparameter tuners you kind of say, okay, here's this objective function. I want you to make this nu…”
The other half of the tape: Josh Albrecht's own voice is left out of every number here. Other people bring the name up 8 times in 1 episode on Latent Space. every mention, with the transcript →
Who brings them up most Kanjun Qiu 8
Every mention by year
tap a year for its mentions
2023 8 mentions in 1 episode
Appearances (1)
| Episode | Date | Speaking time |
|---|---|---|
| State of the Art: Training 70B LLMs on 10,000 H100 clusters | Jun 25, 2024 | 40m |