Parikh: Make-A-Video initializes from pretrained image model parameters before learning motion
Devi Parikh · No Priors Ep. 24 | With Devi Parikh from Meta · Jul 20, 2023 · at 10:54
Devi Parikh details the architecture and training pipeline of Meta's Make-A-Video generative AI model.
0:00 / 0:18exact quote · 18.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Concretely the way it works is that when you initialize the model, you're starting off with image generation sort of parameters that have already been learned. So before you do any training for Make a Video, you're, we set it up so that it can generate a few frames that are Not temporally coherent.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Devi Parikh
Parikh: Generative video progress will lag behind LLM and image advancements
“I think that might be harder in video and I wonder if there is something that we are kind of fundamentally missing in terms of how we approach video generation. So it's not quite answering what you asked me, but I do think that it might be a little bit slower …”
Parikh: Separating motion from appearance enables video models to use uncaptioned data
“There are sort of multiple advantages of thinking of it that way. One is there's less for the model to learn because you're directly bringing in everything that you already know about images to start with. The second is all of the diversity that we have in our…”
Parikh: Generative AI will replace search with direct content synthesis
“Almost everything that you think of images, video, you can ask this question, like for any situation where you're searching for something, trying to find something, It's relevant to ask, well, could I just create what it is that I have in my head? And so when …”
Parikh: Video understanding is more relevant to robotics than video generation
“So I think there the video understanding piece is probably more relevant than the video generation piece.”
Parikh: Generative AI control mechanisms consistently lag core model capabilities
“I think control sort of tends to lag behind the core capability. Like even with images, I feel like we first had to get to a point where these models can actually generate nice looking images before we start worrying about, well, is it really doing what I want…”
Parikh: AI video editing will see faster adoption than from-scratch generation
“We'll probably see much more of, ah, we're already seeing that, and I think we'll see more of where you already have a video that you're starting with, and then you're trying to edit it which has similarities too, but is a little bit different in my mind compa…”