Biderman: Personalized AI adapters require hot-swapping millions of endpoints at inference
Dan Biderman · The AI Memory Problem: Why Long Context Isn’t Enough — Dan Biderman, Engram Co-founder & CEO · Jul 13, 2026 · at 43:43
Dan Biderman, co-founder and CEO of Engram, explains the massive systems and infrastructure challenges involved in building scalable continual learning AI architectures.
0:00 / 0:35exact quote · 35.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And if you truly believe that we can get to the level where we have those kinds of parameter efficient adapters for every person and team, you suddenly think about deployments that involve millions of different endpoints stored in different places that need to be efficiently read from disk to HBM. Yeah. And then use that inference time and swapped and updated. It's going to be, if things work out for us, this thing will have a massive Compute footprint and many new questions on, on systems and balancing of AI workloads in new ways.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dan Biderman
Biderman: AI-native companies will amass trillions of internal tokens within 18 months
“In 18 months, many companies would have maybe trillions of tokens, which of internal company data, proprietary data. I'm talking about like maybe trillions. It sounds exaggerated, but I don't think it's an impossibility if they're really AI native.”
Biderman: Hard engineering tasks will require test-time gradient updates
“We think that eventually part of the solution for very hard tasks in, in science and engineering and defense and all that stuff will involve some form of gradient based updates during during doing these long horizon tasks.”
Biderman: In 18 months, data scale will require weight-based learning
“Other parts of it are bets that in 18 months from now, the scale of the data will require the methods that we know from pre-training work.”
Biderman: PC hardware will soon run near-trillion-parameter models locally
“And in the long, long term, I do think these things will actually run on people's devices, and we're seeing right now the new hardware on personal computers is already, ah, you know, soon approaching the ability to run inference on close to trillion parameters…”
Biderman: Engram trains models to decide what to memorize vs keep in notes
“The way to work on it is to train models both, to train models to manage it themselves, and that's an active area for us. Have the model know, like, without any explicit supervision signal to determine this kind of stuff I can pull from my brain, and that kind…”
Biderman: Manually partitioning LLM memory vs retrieval becomes unmanageable whack-a-mole
“And now the thing is, if you start manually, heuristically saying this is in, this is out, then it becomes a whack-a-mole. Every, every person in every enterprise has different data, and you can really very easily pick and choose what goes in and what goes out…”