The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 30 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 1 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Jeff Dean: Analog computing loses power advantages at digital boundaries
“I mean, I think there's still a, there's also sort of the more exotic things like analog based computing substrates as opposed to digital ones.
I'm, you know, I think those are super interesting cause they can be potentially low power.
but I think you often …”
Dean: General AI models will win out over specialized ones
“I mean, I think general models will win out over specialized ones in most cases.”
Jeff Dean: Capable small models require first building frontier models
“Through distillation, which is a key technique for making the smaller models more capable, you know, you have to have the frontier model in order to then distill it into your smaller model. So it's not like an either or choice. You sort of need that in order t…”
Jeff Dean: Teacher model logits enable small models to learn from multi-pass training
“One of the key advantages of distillation is that you can have a much smaller model And you can have a very large you know, training data set and you can get utility out of making many passes over that data set because you're now getting the logits from the mu…”
Dean: Next-gen Gemini Flash matches or beats prior-gen Gemini Pro
“For multiple Gemini generations now, we've been able to make the sort of flash version of the next generation as good or even substantially better than the previous generations pro, and I think we're gonna keep trying to do that because that seems like a good …”
Jeff Dean: Low latency is critical as AI shifts to complex multi-token tasks
“Latency is actually a pretty important characteristic for these models, because we're gonna want Models to do much more complicated things that are going to involve, you know, generating many more tokens from when you ask the model to do something until it act…”
Dean: Scaling quadratic attention cannot reach billion- or trillion-token context windows
“But that's not going to be solved by purely scaling the existing solutions, which are quadratic. So a million tokens kind of pushes what you can do. You're not going to do that to a trillion tokens, let alone, you know, a billion tokens, let alone a trillion.”
Jeff Dean: Gemini was designed to ingest Waymo LIDAR and robotics telemetry
“I think one of the things about Gemini's multimodal aspects is we've always wanted it to be multimodal from the start. And so, you know, that sometimes to people means text and images and video sort of human-like and audio, audio, human-like modalities, but I …”
Jeff Dean: LLM search funnels trillions of tokens down to 100 documents
“And I think an LLM based system is not going to be that dissimilar, right? You're going to tend to trillions of tokens, but you're going to want to identify, you know, what are the 30,000 ish documents that are with the, you know maybe Thirty million interesti…”
Jeff Dean: Design systems to scale 5x to 10x, not 100x
“And I think a good design principle is you're going to want to design a system so that the most important characteristics could scale by like factors of five or 10, but probably not beyond that, because often what happens is if you design a system for X and so…”
Dean: Applying RL to non-verifiable domains would dramatically improve AI models
“How do you get RL to work for non-verifiable domains? I think it's a pretty interesting open problem because I think that would broaden out the capabilities of the models, the improvements that you're seeing in both math and coding if we could apply those to o…”
Dean: Adding training data for hundreds of languages displaces other model capabilities
“We're always making these kind of you know, trade-offs in the data mix that we train the base Gemini models on. You know, we'd love to include Data from 200 more languages and as much data as we have for those languages. But that's going to displace some other…”
Dean: Sparse models offer 10x compute cost efficiency over dense models
“That gave you like a 10 X improvement in, you know, time to quality. Or compute cost to a given quality level relative to non-sparse models.”
Dean: A one-page internal memo sparked the Gemini unification effort
“I actually wrote a one-page memo saying we were being stupid by fragmenting our resources. So in particular at the time we had you know efforts within Google research on and in the brain team in particular on large language models. We also had efforts on multi…”
Dean: Software development will shift to managing independent AI agent teams
“And so I do think there's going to be more of a style of having lots of independent software agents off doing things on your behalf and figuring out the right sort of human computer interaction model and UI and so on for, When should it interrupt you and say, …”
Jeff Dean: Crisply specifying requirements will become a critical engineering skill
“And the better you get at interacting with these models, And I think one of the ways people will get better is they will get really good at crisply specifying things rather than leaving things to ambiguity. And that is actually probably not a bad thing. It's n…”
Dean: AI systems will achieve 20x to 50x lower latency
“And I think, you know, in the future we'll see models that are, and underlying software and hardware systems that are 20 x lower latency than what we have today, 50 x lower latency.”
Dean: Personalized models with full personal context will beat generic models
“A personalized model that knows you and knows all your state and is able to retrieve overall state you have access to that you opt into is going to be incredibly useful compared to a more generic model that doesn't have access to that. So like, can something a…”
Dean: AI model demand is non-stationary because increased capabilities expand user requests
“I mean, I think that's true if your distribution of what people are asking people the models to do is stationary, right? But I think what often happens is as the models become more capable, people ask them to do more, right?”
Dean: AI benchmarks above 95% accuracy offer diminishing returns due to data leakage
“I think once it hits kind of 95% or something, you get very diminishing returns from really focusing on that benchmark because it's sort of, it's either the case that you've now achieved that capability or there's also the issue of leakage in public data or ve…”
Dean: Single needle-in-a-haystack benchmark is saturated up to 128k context lengths
“As you say that needed single needle in a haystack Benchmark is really saturated for at least context lengths up to one 28 K or something.”
Dean: Accelerator batching is driven by 1000x SRAM data movement energy costs
“And so, all of a sudden, this is why your accelerators require batching, because if you move, like, say, the parameter of a model from SRAM on the chip into the multiplier unit, that's gonna cost you a thousand PicoTools, so you better make use of that, that t…”
Jeff Dean: ML chip design requires predicting research workloads 2-6 years out
“As a hardware designer for ML in particular, you're trying to design a chip starting today
And that design might take two years before it even lands in a data center, and then it has to sort of be a reasonable lifetime of the chip to take you three, four, or f…”
Dean: Memorizing retrievable facts wastes model parameter space
“Having the model devote precious parameter space to remembering obscure facts that could be looked up is actually not the best use of that parameter space, right? Like you might prefer something that is more generally useful in more settings than this obscure …”
Dean: Transformers delivered 10x to 100x compute efficiency over LSTMs
“Transformers similarly gave you a 10 X to a hundred X improvement in, you know compute cost to a given quality level versus say LSTMs at the time.”
Jeff Dean: Distillation originated to compress 50-model ensembles into serviceable form
“Distillation was originally motivated because we were seeing that we had a very large image data set at the time, you know, three hundred million images that we could train on with, you know, I forget, like 20,000 categories or something, so much bigger than I…”
Dean: Google evaluates models against held-out internal benchmarks absent from training data
“So we have a bunch of held out internal benchmarks that we really look at where we know That wasn't represented in the training data at all. There are capabilities that we want the model to have that it doesn't have now, and then we can work on, you know, asse…”
Dean: Google moved its entire search index into memory in 2001
“So in 2001, we introduced we put our entire index in memory. And what that enabled from a quality perspective was amazing, because before, you had to be really careful about You know, how many different terms you looked at for a query, because every one of the…”
Dean: Google Search index updates improved from monthly to sub-minute
“So the update rate actually is the parameter that changed the most. Surprisingly. So it used to be once a month. And then we went to a system that could update any particular page in, like, sub one minute.”
Dean: Early 2B-parameter Google Brain model cut ImageNet-22K error by 70%
“It's two billion parameters vision model trained on 16,000 CPU cores for like multiple weeks. And that's what gave us really good. It gave us a 70% relative error improvement in image net 22 K, which is the 22,000 category thing.”