The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 56 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 1 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Ethan He: Pipeline bug fixes drive more model gains than new algorithms
“And often I find that this is kind of boring, but like a lot of the improvements does not come from new algorithms. It comes from finding small bugs here and there in the data pipeline, in the model training pipeline. Those gave the biggest boost to the model …”
Ethan He: Falling inference costs will enable generative UIs for everything
“So I think as a inference cost come down, we are going to have generative UI for everything.”
Ethan He: Training video models costs roughly the same as medium-scale LLMs
“So surprisingly video models is like the cost is very, is comparable to language models. And obviously the largest scale is language model. Maybe like a medium scale language models.”
Ethan He: Visual intelligence in video generation models stems primarily from language models
“The visual intelligence are actually mostly coming from language. Like, these video models, especially from now, since the diffusion model technology is more mature, the, like, every time you see there, there's some improvement on these models, I would say mos…”
Ethan He: LLM Video Agents Will Orchestrate Diffusion Models and Editing Tools
“Video agents, mostly language models, they'll call these generative model, either it's a separate model or a diffusion head or whatever as tool. So this model can iteratively Refine the results or even like you generate longer content through a very long trend…”
Ethan He: Video Agents Will Reach Production-Grade Quality by Year-End
“I guess by the end of this year is this is going to be a big hit. So the inflection point will be there and the videos generated by video agents can get to like production great quality. So it can be presented and it can be distributed in, in ads.”
Ethan He: Peak pre-training learning rate works best for MoE upcycling
“We found that the best is to use the original highest peak learning rate from pre-training, which works the best.”
Ethan He: Small xAI Team Built Grok Imagine in Three Months
“There were no, no infra, no data, and no model. And it just a few engineers, we built it in three months and released the first model, Grok Imagine,”
Ethan He: Video models require image foundations and 100% synthetic caption pairs
“Building a video model. You actually need to build a image model first and building, building these two models. The data you need is a hundred percent synthetic pair of language and image or language to video because on the internet, actually the videos Don't …”
Ethan He: Neural OS models can synthesize novel user interfaces
“So if you train your neural OS or neural computer on the standard screen recordings on the entire internet, the model can imagine completely new interface to interact with the computer.”
Ethan He: Storing and moving video datasets costs millions per month
“So, so it's like just storing, storing the network, those costs, it's just I guess it would be a few millions per month to just storing everything, not to mention the GPU costs.”
Ethan He: Grok Imagine Video Extension Tracks Full Historical Context
“So the Glock Imagine video extension, it has historical context of all of the previous generated videos. It can it has a context of who is speaking and what objects have appeared and everything having that to generate the next video.”
Ethan He: Long context management in video models leads LLM context work
“I feel this is actually, this part of long contacts is a little bit ahead of the LLM part.”
Ethan He: Powerful video AI will naturally learn to control physical robots
“Once these models can use computers and understand the future state of computer extremely well, the robots might be Might be one of the tools a very powerful AI can use. So the powerful AI might just be able to control the physical embodiment naturally.”
He: Upcycling dense models to MoE beats continuing dense training per FLOP
“By training these upcycled models, you can achieve better accuracy than simply training the dense model further for the same number of flops.”
He: Mixtral's top-k before softmax routing hurts MoE upcycling performance
“We actually found the mix-throughs approach didn't work as well as expected, because the original model, the original switch transformer from Google uses a softmax and topk for a reason. And because of upcycling, if you switch to topk, then softmax, it actuall…”
Ethan He: Video Foundation Models Follow Scaling Laws Like LLMs
“There, once I built the Cosmos one, I realized as this thing also has a scaling law similar to language model.”
Ethan He: Daily iteration speed is the top factor in model training
“When I look at like training models, I don't so actually the top important thing is like how many how many iterations can you do like per, per day? And the more iteration can you do, you can train the model much faster. So if you have a very strong infra and y…”
Ethan He: Coding Models Shift Research Bottlenecks Back to Compute
“Compute might become a bottleneck again, because previously, like if you want to train a new model, say you want to generate new synthetic data and then, or write a new algorithm, it might take a few weeks. And during that period of time, you don't, you might …”
Ethan He: Training generative video models on unlabeled data aids generalization
“For the generative model training, there's also really like a small percentage of unlabeled data. So, so the model is instructed to generate a video without any text instruction. That, that can also help the model generalize.”
Ethan He: Training transformers directly on raw image pixels is impossible
“If you're trying, if you can technically, theoretically train image or video models on pure pixels, but the problem is that the, it's a lot of tokens. So like one image, like it's a thousand by a thousand is like one million tokens, one million pixels. It's im…”
Ethan He: Diffusion transformer training closely mirrors LLM training architecture
“So now the training, training of the diffusion transformer, you already generated models use diffusion transformers. It is actually quite standard. It's very similar to how you train a language transformer models. It's not that much difference. It's just the t…”
Ethan He: Video Models Must Bootstrap From Image Diffusion Models for Semantic Understanding
“After you train such model, such image model, the reason it's a foundation for video models is that image, image models are Cheaper to train and they have much denser connection between language and text. So, sorry, language and images. For example, you train …”
Ethan He: Frame-by-frame video compression enables real-time interactivity, temporal compression adds lag
“That being said, the benefit of the frame per frame compression, we might come back to this later, is real timeliness and interactivity. Because if you strain the output of the model frame by frame, you can As a model can respond to any user request immediatel…”
He: Distillation works because teacher models are simpler than the internet
“I guess the, from the modeling perspective, the strong model, the teacher model is trying to model The image and videos of entire internet. And that distribution is extremely complex. As a step distilled model is just trying to learn from the teacher. The teac…”
He: Grok Imagine 0.9 was first large-scale joint audio-video model deployed
“So Grok Imagine, there were .9, I believe it's is a first first audio video trends model deployed at a large scale.”
Ethan He defines world models as real-time, interactive, long-horizon video
“So word model is like real time, interactive, long horizon videos.”
Ethan He predicts world models will culminate in real-time neural computers
“I think the final state will be, for example, like a video version of Playbook where you can interact with a neural computer. You move your mouse and you click on the generative interface. And it will reply to you through, through pixels generally in real time…”
He: Manual reference video conditioning is a workaround, not true long context
“It doesn't need to have a very long context, but it's, I feel like it's an intermediate solution. It's cheating. Yeah, the model should Be able to like selectively know, like where, where should I draw references?”
Ethan He: RLMs and video models will dynamically pull context like humans
“But humans' contacts can, like, attention can work because we can dynamically pull in contacts from different places. The same mechanism I think it's going to happen for RLMs and video models.”
He: AI first-principles planning calculates the theoretical minimum days to ship
“If you think about some limitation, for example, the current data, like how, how fast can we acquire the videos? And if you think about training the models, like what's the iteration speed? For training a model end-to-end and how, how would adding more GPUs ac…”
He: Elon Musk is very hands-on and works closely with xAI teams
“He also worked very closely with people like people imagine online, like he, he's very hands-on.”
Ethan He: AI watermarking will remain vulnerable to reverse-engineering
“As a limitation is like the technology is, as a paper, Was out there and people can reverse engineer that how to get rid of it. And it's, I think even as it advance, it's still, still possible to reverse engineer it.”
Ethan He: Video Agents Will Transition to Fully Automated Video Production
“So in, in Asian, in Gorky Imagine agent mode, you can still go in there and do, do stuff by yourself. Gradually, as the model capability increase, it will be able to do everything fully automated.”
Ethan He: Language models prompt AI models better than humans
“Most of the people were actually not very good at prompting. Actually, language models have a better sense of how to prompt AI models. AI models know AI models better.”
He: Video Agents Are Inherently Costlier Due to Iterative Multi-Sample Generation
“I think the enterprise will have much more budget for video models because the agents are inherently more expensive than the other video models themselves because they do this iterative process. They generate many, many variations.”
Ethan He left xAI because changing corporate priorities limited LLM research
“For me there's a lot of research you want to do that you cannot do at, as a company. And also like the priorities and objective, the, for company typically can change very fast. It is, it's also the same for XAI. So, so now it's kind of like the time to, there…”
Ethan He: LLMs will soon become context-aware and manage context
“I think one thing pretty, pretty interesting. I think might be happening soon is the language models will be like context aware and manage its own context.”
Ethan He: External heuristic engineering gets absorbed into models
“From our experience, the heuristic engineering also have the models get absorbed into the models themselves.”
Ethan He: Megatron MoE was first to train trillion-parameter MoEs at 40% MFU
“The Megatron MOEs was the first It was the first framework open source to be able to train these MOEs at very large scales, like a hundred billion parameters to even trillion parameters efficiently at like 40% MFU.”
He: Token dropping works in MoE pre-training; dropless excels in fine-tuning
“A lot of pre-training experiments show that Token dropping is very efficient, and it doesn't impact performance, but in some of the, like, the downstream fine-tuning, people realize drop-less is better.”
Ethan He: Hugging Face's sequential GEMM loop for Mixtral is inefficient
“Let's also look at the implementation of Mixtro eight by seven on Hagen-Phys transformer. You will soon notice the, in the expert operation there, You would iterate over all of the experts and compute each of the gem operations one by one. We found that this i…”
Ethan He: MoE experts do not cleanly specialize into semantic domains
“Unfortunately, people didn't find, like, a significant interpretability inside these experts. Say, one expert focus on math, the other focus on literature. I think the problem is that neural network hidden states are already very entangled. So, when hidden sta…”
He: Upcycling a 15B model on 1T tokens yielded 4% MMLU gain
“On other scaling experiments, we tried on 15 B models upcycling and applied on one trillion tokens and achieved roughly about five percent improvement in terms of the validation loss and four percent improvement on MMLU.”
Ethan He: 64 experts is the sweet spot for MoE upcycling
“We found, 64 experts is kind of like the sweet spot. If you increase the number of experts beyond 64, it provides diminishing return.”
Ethan He: NVIDIA spent about a year building the Cosmos model
“One thing I say, like, thanks to my experience at NVIDIA, because first time when we were building Cosmos together, we built it for about a year.”
Ethan He: NVIDIA Cosmos required labelers to describe videos for blind reconstruction
“So that's in the protocol of Cosmos labeling. We required the objective we gave to the labelers was that you have to describe the video as detailed as possible, such that a blind person hears a blob of text, can reconstruct what the video is like from their he…”