The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 23 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 1 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Huang: True AI agents require measurable probability improvements per node
“It's like on each stage of the node, you're gonna have to see a marginal improvement in the probability of success for that particular workload because of non-determinism.”
Huang: Google's internal AI tooling was far superior to competitors
“Google was using AI for systems before everybody else too, right? They invented a transformer, and their internal set of tooling was just so far superior to everything else. Like, it's really hard for people to go back after seeing that.”
Huang: RAG versus fine-tuning is fundamentally just meta-learning
“And like, at the end of the day, it's just all meta-learning, right? Like, all we want is, like, the best meta learning workflow or meta learning setup possible to be able to adapt the model to do anything.”
Huang: PoSE breaks down on needle-in-a-haystack at 500k tokens
“It does start to break down a little bit more on the longer, longer context. So, like, 500,000 to a million it appeared that it doesn't hold as well specifically for, like, needle in the haystack.”
Huang: Adding one billion tokens cannot teach trillion-token models new knowledge
“All models these days are now double-digit trillions, right? So it's kind of a drop in the bucket if you really think I can just put, you know, a billion tokens in there, and I actually think that the model's gonna truly learn new Information.”
Huang: Training CodeLlama on Llama 2 caused catastrophic language forgetting
“We do have historical precedent where CodeLlama was, you know, trained further from the original CodeLlama was trained further from Lama II, and it just lost, All its language capabilities, basically, right?”
Huang: Model merging is polluting open LLM leaderboards
“That is extremely interesting from the developer community, and I want to see more of it except it is, to a certain extent, kind of polluting the leaderboards these days, because it's so targeted, and like, now you can kind of game the metric by just finding a…”
Huang: RULER benchmark is more comprehensive than Gemini's multi-needle test
“I would even argue is more comprehensive than the benchmark that, that Gemini released for their, like, multi-needle in the haystack.”
Huang: Multimodal inputs like video will drive long-context LLM demand
“Multimodality is, in my opinion, going to be, it's going to be pivotal for long context. Just because, like, videos when you're getting into the frames per second and you're getting into lots of images and, like, things that are a lot more, like, embodied You …”
Huang: Correct RoPE theta scaling causes immediate early perplexity drops
“Specifically when the one trick you should pay attention to is you know that your context length and theta scaling is working right if the early steps in the perplexity go straight down. So like when it wasn't correct, it would oscillate a lot in the beginning…”
Huang: DeepSeek's Multi-Head Latent Attention is novel and underrated
“Underrated specific instance would be, like, the deep seek paper. I'd never seen it before, but, like, the multi-head latent attention, like, that was really unexpected to me because, like, I thought I'd seen every Not every type, obviously, but, like, every w…”
Huang: Curriculum context expansion outperforms full-length training from scratch
“If you train a model on a shorter context and you progressively increase that context to, like,
You know, the final limit that you have, like, 32 K is usually the limit of Lama two was that long.
It actually performs better than if you try to train 32 K the …”
Huang: Context scaling requires positional interpolation rather than extrapolation
“There's it's super confusing, but it's like, there's extrapolate positional extrapolation, and then there's interpolation.
You want interpolation.
It's been shown that just pure extrapolation makes the model a lot worse, and it's harder to attend to stuff, whe…”
Huang: Modern LLMs have shifted from ALiBi to RoPE scaling
“Some of the newer architectures don't actually employ it a lot. I think the last architecture that actually really employed it was the Mosaic MPT model class, and then almost all the models these days are all rope scaling, and then effectively you can use yarn…”
Huang: Berkeley's JAX Ring Attention does not work well on GPUs
“The Jaxx implementation just does not work on, on GPUs very well. Like, any naive setup that you do, like, it just won't run out of the box very easily”
Huang: EasyContext was the first functional PyTorch Ring Attention implementation
“Easy context was the first PyTorch implementation that applied it with native libraries that worked pretty well. And then we adapted it ourselves in order to configure it for our cluster network topology.”
Huang: LoRA merging succeeds on style but fails on complex capabilities
“Like, I will not lie to say I'm really surprised how effective it is sometimes, but I do notice that for more complex abilities other than, like, more stylistic stuff, it does, it kind of falls through, because maybe it's, it requires a much deeper path in the…”
Huang: Needle in a Haystack is a primitive LLM benchmark prerequisite
“I think needle in a haystack is definitely, like, the standard for presenting the work in a way that people can understand and also proving out. I would say, like, I view it as, like, a primitive. That you have to pass in order to give the model any shot of do…”
Huang: LLM session state management will require huge context windows
“Making the model track state and have state management over time is really, really hard. And it's an incredibly hard evaluation that will probably only really work when you have a huge context.”
Huang: AI talent race mirrors past quant trading wars
“Now we intersect again when it kind of feels like more or less the same, right? Like the AI wars, the trading wars back in the day too, to a certain extent, and the grab for talent.”
Gradient trained 1M context Llama-3 on Crusoe's Nvidia L40 GPUs
“It just made it really easy for us to, like, scale up with their L-Forties, and those are the specific GPU instances we used, and coordinating that effort with them to get, you know, that dedicated cluster first to do the project”
Huang: Placing answers at the end of long contexts breaks attention
“You could create, like, a long context dataset where, like, every single time the last 200 tokens can answer the entire question, and that's never gonna make the model attend to anything.”
Huang: Gradient aims to transition enterprise RPA to autonomous agents
“Well, quite simply, like, gradient, we're a full stack AI platform, and what we really want to do is we want to enable all of the, you know, RPA workloads or the codified automation workloads that existed in the enterprise before. We really want to enable peop…”