Dec 6, 2025 · 1h 4m · latent-space
World Models & General Intuition: Khosla's largest bet since LLMs & OpenAI
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this extensive studio interview and live technical demonstration, General Intuition CEO Pim de Witte explains how the startup utilizes Medal.tv's 3.8-billion gameplay dataset to build action-conditioned world models and vision-based foundation agents. De Witte details the technical architecture, fundraising journey, and long-term vision of transferring gaming-derived spatial-temporal intelligence into robotics and physical AI.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 5.3 Guest teaching 4.7 Guest disagreement 1.6 The hosts pushing back 1.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 44:04 Pim challenges World Labs 3D splat interactivity
Pim directly challenges whether non-interactive 3D Gaussian splats constitute genuine world models, arguing interactivity is the defining criterion.
Hardest push from the hosts ▶ 45:03 Host presses on 3D physics splats vs 2D frame modelsThe host actively pushes back by citing his interview with Justin Johnson, arguing that physical forces applied to splats yield native 3D interactivity.
Biggest teaching moment ▶ 18:12 Pim explains action abstraction over raw keyboard loggingPim educates on why logging raw keystrokes introduces training noise and privacy liabilities, explaining how mapped action spaces solve both problems.
The host holds their own ▶ 47:10 Host cites Noam Brown on implicit world models in LLMsThe host demonstrates deep technical context by citing Noam Brown's counter-argument regarding implicit world model representations inside autoregressive LLMs.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Defining World Models and Action-State Transitions | 4 | 2 | 1 | 0 | The host delivers an introductory monologue summarizing General Intuition's origin from Medal, comparing data capture mechanics to Tesla bug reports, and detailing the recent $134M seed round. The tone is descriptive and supportive, setting up the interview context. | |
| Live Demonstration: Vision-Only Imitation Learning Gameplay | 5 | 4 | 1 | 2 | Pim walks through screen recordings of the vision-only imitation model navigating FPS environments. The host asks clarifying questions about memory horizon, goal-conditioning, and superhuman versus peak-human performance. | |
| Cross-Domain Action Prediction and Real-World Transfer | 5 | 4 | 2 | 2 | Pim demonstrates cross-game and real-world video transfer of action predictions. The host raises the distinction between egocentric (first-person) versus third-person perspectives, which Pim addresses by explaining multi-object control tradeoffs. | |
| World Models, Physical Dynamics, and Partial Observability | 4 | 5 | 1 | 1 | Pim explains world models handling partial observability (navigating through smoke screens) and inheriting real-world camera shake dynamics. The host tracks the visual demonstration with brief observational confirmations. | |
| Model Distillation and Optical Dynamics of Gameplay | 6 | 5 | 1 | 3 | The host probes the purpose of model distillation if the base model already runs in real time, prompting Pim to explain parameter efficiency. Pim then details how gaming input simulates optical dynamics and reduces information loss compared to YouTube pose estimation. | |
| Medal's 3.8B Dataset and Privacy-Preserving Action Mapping | 4 | 6 | 2 | 1 | Pim explains why Medal maps abstracted game actions instead of raw keystrokes (W/A/S/D) to protect user privacy while avoiding noisy training tokens. The host inquires into the origin of this insight and the manual labeling effort. | |
| Medal's Product Mechanics and Selective High-Value Recording | 5 | 4 | 1 | 2 | Pim details how Medal succeeded by focusing on lightweight retroactive in-memory recording rather than a heavy streaming suite. The host connects this to active learning and Tesla's selective data harvesting for corner cases. | |
| Technical Genesis: DIAMOND, SIMA, and Lab Recruitment | 6 | 5 | 2 | 3 | The host questions the tractability of unbounded continuous action spaces when switching from keyboard/mouse to general actions. Pim explains starting with discrete controller spaces before moving to action embeddings. | |
| GI's Strategic Moat and Environment Diversity | 5 | 5 | 2 | 3 | The host asks whether Meta Quest data could replicate Medal's moat. Pim pushes back by highlighting the necessity of public social graph permissions and the vast diversity disparity between PC game catalogs and VR environments. | |
| Frontier Research: Gaia-2, DIAMOND, and SIMA-2 Analysis | 6 | 4 | 1 | 2 | The host cites recent paper club reviews on Gaia-2, SIMA-2, and Genie-3. Pim breaks down SIMA-2's steerability and how an orchestrator VLM like Gemini can act as a high-level policy supervisor. | |
| Inside Vinod Khosla's $134M Seed Investment Process | 5 | 6 | 2 | 1 | Pim recounts Vinod Khosla's rigorous 2030 backward induction pitch process and shares practical advice on valuing proprietary data in AI deals. The host guides the reflection on commercial valuation and lab negotiations. | |
| Mastering Machine Learning Fundamentals: The Fleuret Course | 5 | 6 | 1 | 1 | Pim shares his personal learning journey through Francois Fleuret's deep learning coursework to master mathematical fundamentals. The host affirms the first-principles methodology. | |
| Contrasting World Model Architectures: World Labs vs. GI | 7 | 4 | 3 | 4 | The host directly contrasts GI's frame-based generation with World Labs' 3D Gaussian splat representation, citing his interview with Justin Johnson. Pim critiques non-interactive splats as incomplete world models while noting their representational merits. | |
| Spatial-Temporal Intelligence vs. Autoregressive Text Backbones | 6 | 5 | 3 | 3 | The host and guest debate whether LLMs inherently build world models or if autoregressive text tokens fail to capture continuous spatial-temporal dynamics. Pim positions text as a high-level compression layer rather than the foundational substrate. | |
| Commercial Deployment: Game Engine Bot APIs | 5 | 4 | 1 | 2 | Pim outlines GI's commercial model: providing game developers with an API to replace deterministic behavior trees with human-like bot policies to maintain player retention during off-peak hours. | |
| Cross-Domain Simulation: Truck Simulators, Self-Driving, and Robotics | 6 | 5 | 2 | 2 | Pim notes Medal has more steering wheel hours logged than Waymo fleet miles, positioning simulated driving as an active learning pre-training ground for robotics. The host validates the relevance of active learning and edge-case harvesting. | |
| RL Transition via Playable Clips and Episodic Memory | 5 | 6 | 1 | 1 | Pim articulates the transition from imitation learning to reinforcement learning by converting Medal's recorded episodic highlights into interactive, playable world model environments with reward modeling. | |
| Open Research Partnerships and the 2030 Physical AI Vision | 6 | 4 | 1 | 1 | Pim announces open research partnerships with Kyutai in Paris and outlines the 2030 target of powering 80% of atoms-to-atoms physical AI tasks. The host connects this to Chan Zuckerberg's virtual biology simulation initiatives in closing. |
Statements from this episode (34)
General Intuition's foundation agent runs purely on vision without reinforcement learning
“This is just a base model. There's no RL, no fine tuning. This model sees no game states. It is purely capable, not sequence acceptance. It's purely predicting the actions from the phrase. That's it.”
The baseline of General Intuition's training dataset is peak human gameplay
“The baseline of our data set is PQ and performance.”
General Intuition action models turn internet videos into free training data
“We transferred it over to a real world video, which means that you can use any video on the internet as free training.”
Third-person vision models will be key for controlling multiple environment objects
“The third person I think will be very, very helpful if you're, for instance, trying to control multiple objects in an environment later on. Right now, I think having fully in perception first person is quite helpful.”
General Intuition pre-trains from scratch and fine-tunes open-source video models
“We made the decision to Pre-trained world models from scratch, but also we've actually been able to fine tune open source video models to get a better sense of physical transfer.”
General Intuition world models support high-speed mouse sensitivity capabilities
“Our world models have mouse sensitivity, which is something that, like, gamers absolutely want, right? So you can have these, like, very rapid movements, which you couldn't do in any other world model.”
Gameplay footage teaches spatial reasoning better than YouTube videos, says de Witte
“When you're playing video games, you're actually simulating the optical dynamics with your hand, right? And I think that, like, that's why I think why games are a better representation of switch support reasoning initially than YouTube videos, for instance.”
Medal holds the internet's largest action-labeled video dataset by orders of magnitude
“We have sort of the largest data set of ground truth action labeled video footage on the internet by maybe one or two orders of magnitude.”
Medal employed thousands of humans to label game actions over 18 months
“We had thousands of humans label every single action you can take in every single video game over the past year and a half which is an enormous amount of action labels.”
Medal successfully bootstrapped its social network by prioritizing single-player capture tools
“A lot of our competitors were focused on solving the social network and the recorder at the same time, and that never, like, our bet was really that we could get so many people to record with us that we could bootstrap the network on top of that, and that work…”
General Intuition will initially model game controller inputs before general action embeddings
“So I think we're going to start with anything you can control using a game controller. But yeah, long term, we want to actually predict maybe like action embeddings and have models sit inside a general action space to be able to transfer out to other inputs as…”
Frontier AI research labs aggressively tried to acquire or recruit Medal's team
“So like, right when that happened, a lot of the labs also started on, started understanding what we had. And so we started very aggressively, multiple labs tried to bring us in, in various ways.”
PC gaming offers tens of thousands of scaled environments compared to VR
“The amount of environments in VR that are, that, that have, like, consumption at scale is probably in, like, the hundreds.
whereas on PC, it's probably in the tens of thousands.”
Lead researchers from the GAIA-2 and DIAMOND papers joined General Intuition
“Anthony Tu, who led the research on Gaia II is, is also one of the engineers that joined our team. So it's all the Diamond the core contributors for Diamond, and then Anthony and we just had three more researchers joining this week.”
General Intuition's world models will initially be orchestrated by Vision-Language Models
“I think our models will initially be used as like you'll have like an orchestrator VLM of sorts. That's kind of like managing instances and instructing them.”
Vinod Khosla tests founders by rigorously reverse-engineering their five-year vision
“He asked you to, like, draw a twenty-thirty picture of your company, and I think he just picks And plus five years. Whatever. I don't know. I did the same to you. Yeah. He asks you to like walk that back from first principles all the way from today. And he ask…”
Proprietary data cannot be accurately valued without training models to evaluate capabilities
“I don't think you can value it unless you actually model it yourself and see what the capabilities are.”
Founders negotiating large AI data deals should demand equity, says de Witte
“I would recommend, if you're gonna do large data deals, like, just try to get like a large chunk of equity in the company that you're doing it with if you can.”
World models are significantly more complex than traditional generative video models
“What world models do is they actually have to understand the full range of possibilities and outcomes from the current states and based on the action that you take generates the next states, right? So the next frame. And so it is a much more sort of complex pr…”
Physics simulation compute complexity scales rapidly with agents, freedoms, and information
“The compute complexity of simulation goes up really, really rapidly with three variables. First, the numbers of agents in an environment. Second they're DOF. So they're individuals of freedom. Yeah. And then third, the information that each action reveals.”
General Intuition is betting maximally on video-to-action transfer over physics simulation
“And so I think for us, it is more so about making a maximal bet on video transfer and interacting with things that are difficult to simulate. And the steerability is also really interesting with text than it is on betting against simulation or something like t…”
De Witte: Static 3D environments without interactivity are not true world models
“However, my understanding is they're currently not interactive, which in my opinion is like the whole point of world models, right? It's environments. They're great environments. And I think from a business perspective, I think they picked a really important p…”
Yann LeCun calling LLMs a dead end inspired General Intuition's founding
“Honestly, Jan's podcast that he did, I don't remember which one it was, but a long time ago, where he basically proclaimed LLMs to be a dead end it was one of the things that inspired me to do this.”
Text and language function fundamentally as compression methods for 3D perception
“The way I think about it is, at Siemens, right, we had sort of a three-dimensional world, then we invented text. As like a, in a way, a compression method, right? So you had, we invented text in order to communicate with each other in a common way, in a way th…”
Text and speech generation will become actions emitted by world models
“I think text prediction is just one of the actions that is going to come out of these, you know, these policies and world models. I think speech and text generation will just be one of the actions that, that can be a part of that.”
Language model labs and world model labs will ultimately converge on capabilities
“I think that there will just be labs coming at this problem from both sides. And everyone ends up in roughly the same place, and the same place will be whatever people think is cool.”
Player retention in scaled video games heavily depends on automated bot quality
“So if you're a game developer, how well you're actually retaining players, it's like if you have a game that's already at scale, it's like decently dependent on how good your bots are.”
Medal has more concurrent sim-drivers than Waymo has autonomous cars
“We have more people at any given time on metal playing with steering wheels and like truck simulator and these types of games than Waymo has cars on the road. It's a ridiculous stat, but it's true.”
General Intuition's initial commercial business model will be an API
“Our business model is initially going to be an API, again, like the Anthropic API”
Making billions of gameplay clips playable bridges imitation learning to reinforcement learning
“Actually making every single clip on the platform playable at billions of clips scale is how we go from imitation learning to RL.”
General Intuition publishes openly because its proprietary data moat prevents model replication
“Because we have such a large data mode, we don't have to be as concerned as the LLM companies about publishing, because we don't need the ones to be able to. Exactly. No one can replicate the models, right?”
General Intuition partnered with Eric Schmidt-backed Kyutai for open data research
“We just announced our partnership with Qtai in France, which is an open science lab in Paris, one of the best research labs in the world. Eric Schmidt, I believe funded in addition to some French people. They are essentially acting as the partner that's curren…”
General Intuition aims to drive 80% of AI physical interactions by 2030
“In the atoms to atoms stage, I want, like, I want GI models to be responsible for 80% of all the atoms to atoms interactions driven by AI models and the reason for that is because we were able to unblock intelligence so quickly, and robotics, like, intelligenc…”
Simulation will initially be a larger commercial AI market than physical deployment
“I think simulation will actually be the larger market initially. So I think in simulation because you have very little constraints also from a safety perspective, simulation is much easier. So I think a lot of the takeoff initially sits in simulation.”