Parikh: Separating motion from appearance enables video models to use uncaptioned data
Devi Parikh · No Priors Ep. 24 | With Devi Parikh from Meta · Jul 20, 2023 · at 9:59
Devi Parikh, Research Director in Generative AI at Meta, explains the architectural advantages of Meta's Make-A-Video system over end-to-end video-text training.
0:00 / 0:47exact quote · 47.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“There are sort of multiple advantages of thinking of it that way. One is there's less for the model to learn because you're directly bringing in everything that you already know about images to start with. The second is all of the diversity that we have in our image data sets. That image models already know all sorts of fantastical depictions of like dragons and unicorns and things like that, but you may not have as much video data easily available. All of that is inherited. So all the diversity of the visual concepts can come in through images, even if your video data set doesn't have all of that. And the third benefit maybe the biggest one is that because of the separation, you don't need video and associated text as per data. You have images and text as per data, and then you just have one label video. To learn motion from.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Devi Parikh
Parikh: Generative video progress will lag behind LLM and image advancements
“I think that might be harder in video and I wonder if there is something that we are kind of fundamentally missing in terms of how we approach video generation. So it's not quite answering what you asked me, but I do think that it might be a little bit slower …”
Parikh: Generative AI will replace search with direct content synthesis
“Almost everything that you think of images, video, you can ask this question, like for any situation where you're searching for something, trying to find something, It's relevant to ask, well, could I just create what it is that I have in my head? And so when …”
Parikh: Video understanding is more relevant to robotics than video generation
“So I think there the video understanding piece is probably more relevant than the video generation piece.”
Parikh: Generative AI control mechanisms consistently lag core model capabilities
“I think control sort of tends to lag behind the core capability. Like even with images, I feel like we first had to get to a point where these models can actually generate nice looking images before we start worrying about, well, is it really doing what I want…”
Parikh: AI video editing will see faster adoption than from-scratch generation
“We'll probably see much more of, ah, we're already seeing that, and I think we'll see more of where you already have a video that you're starting with, and then you're trying to edit it which has similarities too, but is a little bit different in my mind compa…”