Justin Johnson: Multimodal LLMs shoehorn visual data into 1D token sequences
Justin Johnson · “The Future of AI is Here” — Fei-Fei Li Unveils the Next Frontier of AI · Sep 20, 2024 · at 27:44
Justin Johnson, co-founder of World Labs, discusses the structural limitations of using 1D language model sequence architectures for spatial and visual AI reasoning.
0:00 / 0:18exact quote · 18.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And now the multimodal LLMs that we're seeing now, you kind of end up shoehorning the other modalities into this underlying representation of a one D sequence of tokens. Now when we move to spatial intelligence, it's kind of going the other way. Where we're saying that the three-dimensional nature of the world should be front and center in the representation.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Justin Johnson
Justin Johnson: Native 3D AI representations will outperform 2D video generation
“Modeling the two D projections of a dynamic three D world is, is a function that probably can be modeled, but by putting a three D representation into the heart of a model, there's just going to be a better fit between the kind of representation that the model…”
Justin Johnson: Seamless mixed reality will deprecate phones, TVs, and monitors
“If you've got the ability to seamlessly blend virtual content with the physical world, it kind of deprecates the need for all of those.”
Johnson: No base model has used new view prediction before Atlas
“And this is a really fundamental primitive that we think is super exciting, a super new primitive For base models that no one's ever done before, right?”
Johnson: Scaling spatial world models is primarily bottlenecked by training compute
“I think we're basically at the beginning, and we're basically limited by compute at this point, right? Like, data is very important, as Fei-Fei likes to point out, but, like, everything has a bottleneck, and I think the main bottleneck on continuing to scale t…”
Johnson: Generative new view prediction in Atlas is AI-complete
“But I think that's something we're kind of realizing, and Ben was talking about this earlier today, is, like, new view prediction, this primitive that we have in Atlas, especially generative new view prediction, this is also AI complete.”