Sun: Synthetic data matches real-world data for multimodal model pre-training
Fan-yun Sun · Moonlake: Interactive, Multimodal World Models — with Chris Manning and Fan-yun Sun · Apr 2, 2026 · at 2:56
Moonlake co-founder Fan-yun Sun describes the findings from foundation model pre-training experiments conducted during his time with NVIDIA Research.
0:00 / 0:09exact quote · 9.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We were actually generating a lot of synthetic data and showing that, hey, you can actually, these synthetic data are actually as useful as real-world data when it comes to multimodal pre-training.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Fan-yun Sun
Sun: Neural rendering with world priors will replace rasterizers and DLSS
“We actually believe that this is going to be the next paradigm of rendering. So it's going to replace how rasterizers, it's going to replace DLSS today because it not only has these pixel prior that's learned from the world, such that you can literally play an…”
Sun: Pixel-coherent world simulators are overrated for causal reasoning and embodied AI
“Having a world simulator that can produce pixel coherency is very, very useful for games and, you know, marketing and all these things, but it's not as useful as people think when it comes to causal reasoning, when it comes to embodied AI.”
Sun: Embodied General Intelligence Requires Interactive Causal Data
“On our way to, let's call it embodied general intelligence, Models need to learn the consequences behind their actions, which means that they need interactive data.”
Sun: Using structural abstraction in AI does not contradict Bitter Lesson
“I do feel like sometimes people confuse like, oh, like we're taking an, a method with abstraction. That means they don't believe in bitter lesson. Like that's just false, right? Like we are believers in bitter lesson, but then I feel like the question that we …”
Sun: Moonlake splits world modeling into multimodal reasoning and Reverie rendering
“Within our world modeling framework, we think there are two models that we train, right? Like there's the multimodal reasoning model that we just talked about that essentially handles Mainly the causality, the persistency, and logic, determinism, determinism o…”
Sun: Code modifies physical rules better than purely data-driven models
“But it's a lot easier to change with code, as opposed to a model that is Learned primarily on data of real world and virtual worlds that are, I guess, like for example, junior, like there's actually trained on a lot of real world data and a lot of virtual gami…”