The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 14 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Schmidt: DisTrO will eliminate data centers' InfiniBand dependency before edge AI dominates
“I think there's, you know, immediately coming out. What you'll see is the ability for even centralized actors who might have multiple data centers to now, like just use them in a more efficient way. Like just have N equals two, you know, like anything. And eac…”
Schmidt: Decentralized training of 400B parameter AI models is solvable by 2025
“I think it still is, it would still be, you know, like a next year sort of environment thing that we would have to do. There are some scaling problems, or not scaling problems, but technical things about how you shard the model, because at that point you get t…”
Schmidt: AI industry relies on outdated 1990s architecture assumptions ripe for disruption
“It turns out that most assumptions in the AI space right now are a product of that's just how things had been done when there was not nearly as much energy and attention to it. So someone made an assumption maybe in the early nineties that everyone just kind o…”
Schmidt: Relying solely on fine-tuning is an existential threat to open-source AI
“Because that for us, if we're just fine tuning models. Right. That's like an existential threat, right? That's like an actual existential threat because the closed providers will continue to get better and we would be like dead in the water in a lot of sense.”
Schmidt: Sharing only key signals in distributed training yields equivalent model learning
“We know that like what we, what needs to be communicated between these things, the two, the different nodes are just these few key pieces of information. And that is necessary. That is a necessary condition or rather a sufficient condition To get the equivalen…”
DisTrO trains multiple models in a bounded search space instead of full synchronization
“So with distro, what we found is that rather than bringing everyone back home and averaging it back together, what you want to do is give each of those little nodes that are searching for the lowest point in the lost landscape, the freedom to move around. And …”
Jeff Schmidt: Hermes pioneered synthetic data training before it was standard
“So Hermes was very early to the idea that you could have synthetic data, which is that you could actually make, you could make a better model by taking an AI model, having it generate words and text, and then training a new a model on that output. This is now …”
Schmidt: Nous is building fault-tolerant training code for heterogeneous devices
“We're making sure that also like the code we're writing to help to actually do this training is agnostic to the hardware and is able to communicate and operate. You can have an Apple device and an NVIDIA device training together. And this is actually the, for …”
Schmidt: ChatGPT, Llama, and DeepSeek use Nous Research's YaRN context extension
“Bone here is the lead author of a method we developed called YARN, which is a context window extension method that we released and did the research on. It is now used by every, every model you use nowadays, everything, everything Chachipiti, Lama, DeepSeq, all…”
Schmidt: Fewer than ten organizations worldwide can train Llama-scale AI models
“Yeah, I mean, I would, it would probably be in the number of ones on my hand and it probably wouldn't use all my fingers, you know. Yeah, I mean, you basically have, OpenA, Anthropic, Meta, X, Google, and then you have a few Mistral, and then Deep Seek and a c…”
Nous Research replicates DisTrO training results using Allen AI's OLMo framework
“And we've re-implemented now a third time in their framework, and we're able to reproduce their training run exactly, and then did it again with Distro, got the exact same results we got with Natron and stuff.”
Schmidt: Zeroth-Order Optimization Requires 1,000x More Computation Than Backpropagation
“What we discovered is that backprop is still being like, you really do still need to be doing back propagation to find the optimal point of the loss. And it's just like zeroth order is like, what, like a thousand, like it worked, but it was like, you needed li…”