Jun 1, 2026 · 1h 44m · latent-space
Inside xAI: Building Grok Imagine in 3 Months, Videogen vs World Models, and Video Agents— Ethan He
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this technical interview from Latent Space, former xAI and NVIDIA researcher Ethan He breaks down the architectural foundations of video foundation models, the creation of Grok Imagine, and the defining characteristics of interactive world models. He argues that true visual intelligence is driven primarily by large language model reasoning, laying out a roadmap for how autonomous video agents and real-time generative interfaces will transform media production and computing.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 18.2% of the talking time here. How this is scored →
The hosts as informed peer 5.3 Guest teaching 5.1 Guest disagreement 1.3 The hosts pushing back 1.8
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:14:48 Visual intelligence comes from language
Ethan asserts his contrarian central claim that video diffusion models are inherently limited and nearly all recent visual intelligence advancements stem directly from language models.
Hardest push from the hosts ▶ 28:10 Challenging compute deflation rateSwyx directly rejects Ethan's estimate that compute costs drop 2x per year, asserting that effective language model inference costs drop by 100x to 1000x every 12 to 18 months.
Biggest teaching moment ▶ 1:15:18 Deconstructing prompt upsamplers in diffusionEthan breaks down how naive diffusion models interpret literal prompts poorly, educating the hosts on why large language model upsamplers do the actual heavy lifting of scene reasoning.
The host holds their own ▶ 35:10 Real-time cloud storage and egress pricing lookupSwyx leverages direct infrastructure knowledge by calculating live S3 standard tier and egress transfer costs to demonstrate the multi-million dollar storage realities of video foundation models.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Welcome and Latent Space Community Roots | 4 | 3 | 1 | 1 | Swyx welcomes Ethan He, recalling his past papers presented at the Latent Space paper club. Ethan explains his background working on Cosmos at NVIDIA and transitioning to xAI for compute scaling. | |
| The Three-Month Sprint: Team Dynamics and Iteration Speed | 5 | 5 | 1 | 2 | Swyx asks about the sequence of building a video generative pipeline from scratch in three months. Ethan emphasizes team bandwidth, rapid iteration loops, and finding small bugs in pipelines over inventing new algorithms. | |
| Coding Models and Compute as the Iteration Bottleneck | 4 | 4 | 1 | 1 | Ethan explains how coding models automated implementation, shifting the bottleneck back onto raw compute availability for rapid experimentation. Swyx and Vibhu discuss the cost and pressure of burning cluster compute. | |
| Synthetic Data Pipelines and Detailed Captioning Protocols | 6 | 5 | 2 | 3 | Ethan details synthetic text-video pairing and Cosmos captioning protocols designed for blind reconstruction. Swyx pushes on the difference between supervised dense captioning and modern unsupervised multimodal pretraining. | |
| Tokenization, Latent VAEs, and Diffusion Transformers | 6 | 5 | 1 | 1 | Ethan explains continuous latent spaces via VAEs and visual tokenization for diffusion transformers. Swyx references Vision Transformer patching papers and historical convolution comparisons. | |
| Image Foundation Models as the Semantic Anchor for Video | 6 | 6 | 1 | 2 | Swyx asks why standard video compression like MP4 isn't directly used as tokens. Ethan explains why MP4 representations are hard for transformers to learn and breaks down temporal versus spatial VAE compression tradeoffs. | |
| Generative User Interfaces and Real-Time Interaction | 6 | 4 | 2 | 4 | Ethan and the hosts analyze Flipbook's real-time generative UI paradigm. Swyx challenges Ethan's claim of compute cost dropping 2x annually by pointing out effective language model inference cost drops 100x to 1000x every 12 to 18 months. | |
| Neural OS: Simulating Operating Systems via Video Models | 5 | 5 | 1 | 2 | The hosts discuss Neural OS simulating operating systems in video. Ethan explains that training on internet screen recordings enables neural computers to generalize beyond existing static desktop interfaces. | |
| The Storage and I/O Economics of Video Model Training | 7 | 6 | 1 | 2 | Ethan outlines the petabyte-scale storage and network transfer costs of video training data. Swyx looks up live AWS S3 tiering and egress pricing to validate and demonstrate exact cloud expenditure figures. | |
| Model Architecture: Scaling Parameters and Visual Tokens | 6 | 6 | 1 | 1 | Ethan compares video model parameter scaling and step distillation techniques to language models. Swyx links the step reduction intuition to consistency models and historical GAN dynamics. | |
| Joint Audio-Video Generation in Grok Imagine | 5 | 6 | 1 | 1 | Ethan describes shipping joint audio-video generation in Grok Imagine 0.9, detailing the challenges of continuous audio modeling and strict temporal alignment. | |
| Temporal Grounding and World Understanding in AI Models | 6 | 5 | 2 | 3 | Ethan argues LLMs lack intrinsic time grounding, which Vibhu counters by pointing out that text priors reflect human duration estimates. Swyx introduces recursive world model requirements. | |
| Solving Long-Horizon Video Generation with Video Extension | 5 | 6 | 1 | 2 | Ethan defines world models as real-time, interactive, long-horizon systems and explains how Grok Imagine implemented video extension with full history to avoid cumulative frame degradation. | |
| Reference-to-Video and Character Consistency Systems | 5 | 6 | 1 | 2 | Ethan details reference-to-video conditioning across multiple image inputs for character consistency. Swyx critiques xAI's minimal public communication of its technical breakthroughs. | |
| Dynamic Context Management: Frame Packing and Attention | 6 | 5 | 1 | 2 | Ethan reviews frame packing heuristics and context window management. Swyx connects this to the Claude Code context pruning leak and discusses dynamic attention mechanisms. | |
| xAI Engineering Culture and First-Principles Execution | 4 | 5 | 1 | 1 | Ethan explains xAI's first-principles execution philosophy, calculating minimum physical time requirements for data ingestion and training iterations. Swyx notes Elon Musk's physics-based management style. | |
| Real-Time Interactivity in Grok Voice Mode | 5 | 4 | 1 | 1 | The hosts discuss Grok voice mode performance, SynthID watermark stripping, and visual artifact detection in generative video models. | |
| The Core Thesis: Visual Intelligence Originates from Language | 5 | 7 | 3 | 1 | Ethan states his provocative central thesis: visual intelligence originates from language models rather than the diffusion models themselves, explaining how prompt upsamplers supply compositional reasoning. | |
| Multimodal Reasoning Paradigms and External Tool Orchestration | 6 | 5 | 1 | 2 | The group contrasts unified Omni architectures with separate prompt rewriter and diffusion head pipelines. Swyx clarifies differences between autoregressive language models with diffusion heads versus standalone systems. | |
| Grok Imagine Agent Mode and Long-Form Video Automation | 4 | 5 | 1 | 1 | Ethan discusses Grok Imagine Agent mode and how video generation expands into multi-step agentic workflows combining diffusion with editing operations. | |
| The Video Agents Thesis: Software Harnesses over Pure Generation | 6 | 5 | 2 | 3 | Swyx expresses disappointment that future gains rely on software harnesses rather than pure foundation model scaling. Ethan clarifies that language agents orchestrating deterministic tools and diffusion heads solve precise creative needs. | |
| Timeline Predictions: Inflection Point for Production Video Agents | 5 | 5 | 1 | 2 | Ethan predicts enterprise production-grade video agents will hit an inflection point within the year. Swyx compares Ethan's focus on video generation to other world model researchers targeting embodied physical robotics. | |
| Why Ethan Left xAI to Focus on Large Language Models | 6 | 5 | 2 | 2 | Ethan reveals why he left xAI to focus squarely on LLMs and self-modifying context harnesses. Swyx terms the realization that media generation relies entirely on language intelligence a 'black pill' for generative media specialists. | |
| Ethan's Career Trajectory: Computer Vision, Scaling, and Language | 5 | 5 | 1 | 1 | Ethan reflects on his career journey from ResNet author collaborations at FAIR and Megatron MoE at NVIDIA to xAI, concluding that core large-scale ML principles make switching domains seamless. |
Statements from this episode (50)
Ethan He: Video Foundation Models Follow Scaling Laws Like LLMs
“There, once I built the Cosmos one, I realized as this thing also has a scaling law similar to language model.”
Ethan He: Small xAI Team Built Grok Imagine in Three Months
“There were no, no infra, no data, and no model. And it just a few engineers, we built it in three months and released the first model, Grok Imagine,”
Ethan He: NVIDIA spent about a year building the Cosmos model
“One thing I say, like, thanks to my experience at NVIDIA, because first time when we were building Cosmos together, we built it for about a year.”
Ethan He: Daily iteration speed is the top factor in model training
“When I look at like training models, I don't so actually the top important thing is like how many how many iterations can you do like per, per day? And the more iteration can you do, you can train the model much faster. So if you have a very strong infra and y…”
Ethan He: Pipeline bug fixes drive more model gains than new algorithms
“And often I find that this is kind of boring, but like a lot of the improvements does not come from new algorithms. It comes from finding small bugs here and there in the data pipeline, in the model training pipeline. Those gave the biggest boost to the model …”
Ethan He: Coding Models Shift Research Bottlenecks Back to Compute
“Compute might become a bottleneck again, because previously, like if you want to train a new model, say you want to generate new synthetic data and then, or write a new algorithm, it might take a few weeks. And during that period of time, you don't, you might …”
Ethan He: Video models require image foundations and 100% synthetic caption pairs
“Building a video model. You actually need to build a image model first and building, building these two models. The data you need is a hundred percent synthetic pair of language and image or language to video because on the internet, actually the videos Don't …”
Ethan He: NVIDIA Cosmos required labelers to describe videos for blind reconstruction
“So that's in the protocol of Cosmos labeling. We required the objective we gave to the labelers was that you have to describe the video as detailed as possible, such that a blind person hears a blob of text, can reconstruct what the video is like from their he…”
Ethan He: Training generative video models on unlabeled data aids generalization
“For the generative model training, there's also really like a small percentage of unlabeled data. So, so the model is instructed to generate a video without any text instruction. That, that can also help the model generalize.”
Ethan He: Training transformers directly on raw image pixels is impossible
“If you're trying, if you can technically, theoretically train image or video models on pure pixels, but the problem is that the, it's a lot of tokens. So like one image, like it's a thousand by a thousand is like one million tokens, one million pixels. It's im…”
Ethan He: Diffusion transformer training closely mirrors LLM training architecture
“So now the training, training of the diffusion transformer, you already generated models use diffusion transformers. It is actually quite standard. It's very similar to how you train a language transformer models. It's not that much difference. It's just the t…”
Ethan He: Video Models Must Bootstrap From Image Diffusion Models for Semantic Understanding
“After you train such model, such image model, the reason it's a foundation for video models is that image, image models are Cheaper to train and they have much denser connection between language and text. So, sorry, language and images. For example, you train …”
Ethan He: Training models directly on MP4 tokens is extremely difficult
“So people actually have tried that, but the main challenge is the latent space for the MP four tokens are not, we're not very comprehensible for the models. It's extremely hard to train on that.”
Ethan He: Temporal video compression cuts context length up to 4x
“The difference is if you compress the temporal dimension, you get a much higher compression rate. Because there is temporal redundancy between frames, because this frame and the last frame, likely they are mostly similar. So there's only some small difference.…”
Ethan He: Frame-by-frame video compression enables real-time interactivity, temporal compression adds lag
“That being said, the benefit of the frame per frame compression, we might come back to this later, is real timeliness and interactivity. Because if you strain the output of the model frame by frame, you can As a model can respond to any user request immediatel…”
Ethan He: Falling inference costs will enable generative UIs for everything
“So I think as a inference cost come down, we are going to have generative UI for everything.”
Shawn Wang: Fixed-capability LLM inference costs drop 100x to 1000x annually
“In language models, it is roughly 100 to a thousand times every 12 to 18 months for the same given level of LMSYS ELO.”
Ethan He: Neural OS models can synthesize novel user interfaces
“So if you train your neural OS or neural computer on the standard screen recordings on the entire internet, the model can imagine completely new interface to interact with the computer.”
Ethan He: Training video models costs roughly the same as medium-scale LLMs
“So surprisingly video models is like the cost is very, is comparable to language models. And obviously the largest scale is language model. Maybe like a medium scale language models.”
Ethan He: Stored continuous VAE latent features match raw video size
“You use a VAE to compress the videos and you also need to store, typically you need to store those continuous feature on also in your storage. That's also comparable size with the videos themselves.”
Ethan He: Storing and moving video datasets costs millions per month
“So, so it's like just storing, storing the network, those costs, it's just I guess it would be a few millions per month to just storing everything, not to mention the GPU costs.”
NVIDIA Cosmos trained on tens of trillions of visual tokens
“And if you look at a number of tokens we disclose that in cosmos, it's also like tens of trillions of tokens. On the visual tokens.”
He: Distillation works because teacher models are simpler than the internet
“I guess the, from the modeling perspective, the strong model, the teacher model is trying to model The image and videos of entire internet. And that distribution is extremely complex. As a step distilled model is just trying to learn from the teacher. The teac…”
He: NVIDIA Cosmos runs in 4 to 8 steps, or 1 step for transfer
“In Cosmos, I believe we have like four steps and eight steps. If you do some simpler task, like image to image translation, it can even run in first step, that one step in, in Cosmos transfer.”
He: Grok Imagine 0.9 was first large-scale joint audio-video model deployed
“So Grok Imagine, there were .9, I believe it's is a first first audio video trends model deployed at a large scale.”
He: Audio-video models require precise time alignment unlike text-to-video
“So one important thing is like the alignment. So the model has to know like the video and audio, the it has to have a time based alignment, like at which time step the video and the audio token correspond to each other. We actually don't have these kind of ali…”
Swyx: A full world model must be recursive and self-aware
“A full world model must also be recursive, meaning that the participant in the world model must also be aware that they have a world model. Which is like this whole recursive thing down the line. But yes, and that the world model can be wrong, and that they ne…”
Ethan He defines world models as real-time, interactive, long-horizon video
“So word model is like real time, interactive, long horizon videos.”
Ethan He predicts world models will culminate in real-time neural computers
“I think the final state will be, for example, like a video version of Playbook where you can interact with a neural computer. You move your mouse and you click on the generative interface. And it will reply to you through, through pixels generally in real time…”
Ethan He: Grok Imagine Video Extension Tracks Full Historical Context
“So the Glock Imagine video extension, it has historical context of all of the previous generated videos. It can it has a context of who is speaking and what objects have appeared and everything having that to generate the next video.”
NVIDIA Cosmos uses 50,000 to 60,000 tokens for five seconds of video
“Yeah, for example, like in Cosmos, I think just five seconds of video is like a 50, 50 K or a 60 K number of tokens. So like, if you do 50 seconds as a 500 K tokens, if you do longer than that, easily explode.”
He: Manual reference video conditioning is a workaround, not true long context
“It doesn't need to have a very long context, but it's, I feel like it's an intermediate solution. It's cheating. Yeah, the model should Be able to like selectively know, like where, where should I draw references?”
Ethan He: Long context management in video models leads LLM context work
“I feel this is actually, this part of long contacts is a little bit ahead of the LLM part.”
Ethan He: RLMs and video models will dynamically pull context like humans
“But humans' contacts can, like, attention can work because we can dynamically pull in contacts from different places. The same mechanism I think it's going to happen for RLMs and video models.”
He: AI first-principles planning calculates the theoretical minimum days to ship
“If you think about some limitation, for example, the current data, like how, how fast can we acquire the videos? And if you think about training the models, like what's the iteration speed? For training a model end-to-end and how, how would adding more GPUs ac…”
He: Elon Musk is very hands-on and works closely with xAI teams
“He also worked very closely with people like people imagine online, like he, he's very hands-on.”
Ethan He: Grok Imagine enforces country-specific watermarking and fast takedowns
“So in all of the, those countries Grok imagined had watermarks and a lot of the, lots of takedowns of It's the videos who are also happening extremely fast.”
Ethan He: AI watermarking will remain vulnerable to reverse-engineering
“As a limitation is like the technology is, as a paper, Was out there and people can reverse engineer that how to get rid of it. And it's, I think even as it advance, it's still, still possible to reverse engineer it.”
Ethan He: Visual intelligence in video generation models stems primarily from language models
“The visual intelligence are actually mostly coming from language. Like, these video models, especially from now, since the diffusion model technology is more mature, the, like, every time you see there, there's some improvement on these models, I would say mos…”
Ethan He: NVIDIA Cosmos uses a 7B video model with a larger LLM rewriter
“I think in in Cosmos, we use Lama or we use mix, mix through. And the Cosmos video model itself is only seven B, and the model, the language model is a prompt rewriter. It's bigger than that.”
Ethan He: LLM Video Agents Will Orchestrate Diffusion Models and Editing Tools
“Video agents, mostly language models, they'll call these generative model, either it's a separate model or a diffusion head or whatever as tool. So this model can iteratively Refine the results or even like you generate longer content through a very long trend…”
Ethan He: Video Agents Will Transition to Fully Automated Video Production
“So in, in Asian, in Gorky Imagine agent mode, you can still go in there and do, do stuff by yourself. Gradually, as the model capability increase, it will be able to do everything fully automated.”
Ethan He: Language models prompt AI models better than humans
“Most of the people were actually not very good at prompting. Actually, language models have a better sense of how to prompt AI models. AI models know AI models better.”
Ethan He: Video Agents Will Reach Production-Grade Quality by Year-End
“I guess by the end of this year is this is going to be a big hit. So the inflection point will be there and the videos generated by video agents can get to like production great quality. So it can be presented and it can be distributed in, in ads.”
He: Video Agents Are Inherently Costlier Due to Iterative Multi-Sample Generation
“I think the enterprise will have much more budget for video models because the agents are inherently more expensive than the other video models themselves because they do this iterative process. They generate many, many variations.”
Ethan He: Powerful video AI will naturally learn to control physical robots
“Once these models can use computers and understand the future state of computer extremely well, the robots might be Might be one of the tools a very powerful AI can use. So the powerful AI might just be able to control the physical embodiment naturally.”
Ethan He left xAI because changing corporate priorities limited LLM research
“For me there's a lot of research you want to do that you cannot do at, as a company. And also like the priorities and objective, the, for company typically can change very fast. It is, it's also the same for XAI. So, so now it's kind of like the time to, there…”
Ethan He: LLMs will soon become context-aware and manage context
“I think one thing pretty, pretty interesting. I think might be happening soon is the language models will be like context aware and manage its own context.”