Biderman: Manually partitioning LLM memory vs retrieval becomes unmanageable whack-a-mole
Dan Biderman · The AI Memory Problem: Why Long Context Isn’t Enough — Dan Biderman, Engram Co-founder & CEO · Jul 13, 2026 · at 31:47
Dan Biderman, CEO of Engram, explains why hardcoded heuristic rules cannot effectively manage parametric versus retrieval memory across diverse enterprise environments.
0:00 / 0:30exact quote · 30.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And now the thing is, if you start manually, heuristically saying this is in, this is out, then it becomes a whack-a-mole. Every, every person in every enterprise has different data, and you can really very easily pick and choose what goes in and what goes out. So the holy grail is have the model learn for itself, have it operate with a notebook where it can take notes, have it operate with a brain, associative parameter efficient thing that it can read from, and have it decide when to go to each, and do this with training in an unconstrained way.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dan Biderman
Biderman: AI-native companies will amass trillions of internal tokens within 18 months
“In 18 months, many companies would have maybe trillions of tokens, which of internal company data, proprietary data. I'm talking about like maybe trillions. It sounds exaggerated, but I don't think it's an impossibility if they're really AI native.”
Biderman: Hard engineering tasks will require test-time gradient updates
“We think that eventually part of the solution for very hard tasks in, in science and engineering and defense and all that stuff will involve some form of gradient based updates during during doing these long horizon tasks.”
Biderman: In 18 months, data scale will require weight-based learning
“Other parts of it are bets that in 18 months from now, the scale of the data will require the methods that we know from pre-training work.”
Biderman: PC hardware will soon run near-trillion-parameter models locally
“And in the long, long term, I do think these things will actually run on people's devices, and we're seeing right now the new hardware on personal computers is already, ah, you know, soon approaching the ability to run inference on close to trillion parameters…”
Biderman: Engram trains models to decide what to memorize vs keep in notes
“The way to work on it is to train models both, to train models to manage it themselves, and that's an active area for us. Have the model know, like, without any explicit supervision signal to determine this kind of stuff I can pull from my brain, and that kind…”
Biderman: AI models must learn to autonomously filter out erroneous user feedback
“Increasingly the models will get better, and increasingly they'll know more things than we do, so the model in some way has to learn and understand and kind of, like, discern what, which feedback is valuable and which feedback should be ignored.”