Josh Albrecht
Co-founder & CTO, Imbue. On 2 shows, 2 appearances. The Shows tab opens the full record on each.
founderexecutiveengineerscientistinvestor@joshalbrecht ↗LinkedIn ↗joshalbrecht.com ↗
Josh Albrecht is the co-founder and CTO of Imbue, an AI research lab developing reasoning-focused AI agents. He previously co-founded AI recruiting startup Sourceress, worked as an infrastructure engineer at Addepar, and is a general partner at Outset Capital.
2shows
2appearances
33statements
5resolved
5supported
0contradicted
100%fully supported
8said about them ↓ Everything Josh Albrecht said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Albrecht: LLM emergence is an artifact of non-linear evaluation metrics
“This emergent behavior that you're seeing, Is not really emergent behavior, but is really a function of the evaluation metrics that we're using.”
Albrecht: Training on AWS prevents diagnosing low-level hardware errors
“And if we're just using, you know, AWS or some other cloud provider, These errors are still going to be there, and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong.”
Albrecht: Benchmark performance differences vanish once ambiguous questions are cleaned
“The main takeaway from any of the, like, actual performance is like, once you fix these ambiguous examples, a lot of these benchmarks are really saturated. Like, I think it's important to look at like, you know, like when you're talking about performance on NL…”
Albrecht: AgentBench paper's appendix examples are actually incorrect solutions
“Like we were looking at the agent bench paper, I think just last week for our paper club. And one of the things that we noticed is that actually like both of the examples in the appendix that are given as like traces where it got it right. This is actually not…”
Albrecht: Optimizing for competitive coding benchmarks does not create useful programmers
“Like, we do a lot of code generation, but we don't really do a lot on, like, code competition problems for the very, very hard ones, so that you can go very far down that route and make something like really good at those problems, but not actually that useful…”
Albrecht: Pure LLMs theoretically cannot learn general multiplication algorithms due to context
“Like, we know even in theoretical senses, like, they cannot learn to do multiplication in the general sense because it literally doesn't fit in the context window, right? Like, multiplication, they can learn to do addition in a modular sense, and they can lear…”
Albrecht: Imbue has sufficient compute to train state-of-the-art sized AI models
“We have enough compute to be able to train models that are as large as the largest models have been trained today to date.”
Albrecht: Imbue avoids Kubernetes to keep cluster infrastructure simple to debug
“Less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes, for example, I would just directly launch these things and it's just been much easier to debug this way.”
Albrecht: Vision is not essential for most coding and reasoning agent tasks
“And actually we found that for most of the kind of like code writing and reasoning problems that we care about, the visual part isn't really a huge important part of it.”
Albrecht: Dynamic data mix schedules during LLM training yield negligible gains
“We did some experiments and we've actually talked to a bunch of researchers who were doing work here as well and looking at kind of their experiments on this. And we were originally pretty hopeful because it sounds like something that should work and make sens…”
Albrecht: Code execution expands agent capabilities far beyond hard-coded tool calling
“Instead of worrying about like weird hard coded agents using tools, Like let's just make them able to actually write code robustly and make that code work and be able to debug that code, know if that code is safe to run, like get really good at the like code w…”
Albrecht: Messy real-world data limits knowledge graphs to niche problems
“But I think in the real world, it gets a lot messier than like knowledge graph style of things where it's like, well, is there a relationship between these two nodes? Like, ah, I don't know. Like is, are these two separate nodes? Like those kinds of messy bord…”
Albrecht: Imbue will hire fewer recruiting coordinators as internal agents handle scheduling
“Now, you know, I think probably within the next year, we'll probably, you know, not be hiring as many recruiting coordinators because, oh, we're going to do some of the scheduling with the agent that we've built, right?”
Albrecht: Imbue's GPU cluster failure rate is well below industry 3% benchmark
“The number that we've heard from other people is like they're having about three percent. I don't think we're experiencing failure rates that are that high. I think ours is actually quite a bit lower than that, probably because we've taken the time to like dig…”
Albrecht: Cost-aware tuning reveals scaling laws for all hyperparameters
“So by doing that, we can see the scaling laws or not just, you know, the scaling laws from like the, you know, chinchilla paper, the scaling laws for all parameters. We can see how does the number of layers change with this? How does the You know, the learning…”
Albrecht: Imbue reproduced 500-1,000 examples per dataset to stop eval contamination
“Let's just reproduce, you know, 500 to a thousand examples for every single one of these data sets ourselves and just make sure that this data is definitely not in the, you know, the training set. So we did that and then we're able to like now be confident abo…”
Albrecht: Coding agents communicating uncertainty are far more useful than slightly more accurate ones
“I would much rather have a coding agent that will give me back a thing. And you know, it's actually the code doesn't work like 10% less of the time than some other model, but it will tell me a hundred percent of the time. When it got like when it's not sure, l…”
Albrecht: The true promise of AI lies in autonomous agents, not chatbots
“Right now, you can ask some kind of chatbot something and it'll give you back a response, but the burden is sort of on you to go do something with that to verify whether it's correct or not. I think the real promise of AI is if we can get systems that can actu…”
Albrecht: Repeatable AI agent workflows must progressively transition into explicit code
“And as you do things that you want to do more robustly and you want to do in a more repeatable way, then you want to move it more towards code, right? And so To the extent that you've never seen this task before, maybe you should be doing it in this more kind …”
Albrecht: Imbue focuses on coding agents because objective evaluation is significantly easier
“One of the reasons why we work on code is that there are objective answers to a lot of these questions, either the test pass or they don't, either the function is correct or it isn't. Those kinds of things are much easier to evaluate.”
Albrecht: Narrow AI agents for email triage will work by late 2024
“Yeah, I think a year from now we're going to start to see some of these use cases actually work that today you could you can write these like we have the capabilities you can make some kind of agent to triage your email or to do scheduling or many of these wor…”
Albrecht: Users will create bespoke agents via natural language within five years
“And I think five years from now we're going to have something where it's not just you know, okay we have a scheduling bot we have this other thing but we really have these more general more robust systems where Each of us can individually say, like, I want a t…”
Albrecht: AI agents will dramatically improve codebase quality across the industry
“I think there'll just be a huge flourishing of much higher quality, better software as a result, not just more software, but just taking the existing software and making it so much better, which will make it so much nicer and more fun to interact with as progr…”
The other half of the tape: Josh Albrecht's own voice is left out of every number here. Other people bring the name up 8 times in 1 episode across the shows. every mention, with the transcript →
Who brings them up most Kanjun Qiu 8
Every mention by year
tap a year for its mentions
Latent Space 8
2023 8 mentions in 1 episode
One line per show, most statements first. The link opens Josh's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
| LATENT SPACELEDGER | Co-founder & CTO, Imbue | 1 | 22 | 100% 4/4 | full record on Latent Space → |
 NO PRIORSLEDGER | Co-founder & CTO, Imbue | 1 | 11 | 100% 1/1 | full record on No Priors → |