The Wisdom Wall

“And often I find that this is kind of boring, but like a lot of the improvements does not come from new algorithms. It comes from finding small bugs here and there in the data pipeline, in the, in the model training pipeline. Those gave, those gave the biggest boost to the model quality.”

“So surprisingly video models is like the, the cost is very, is comparable to language models. And obviously the largest scale is language model. Maybe like a medium scale language models.”

“The, uh, the, the visual intelligence are actually mostly coming from language. Like, these video models, especially from now, since the diffusion model technology is more mature, the, like, every time you see there, there's some improvement on these models, I would say mostly, uh, this again comes from language model,…”

“We found that the best is to use the original highest peak learning rate from pre-training, which works the best.”

“building a video model. You actually need to build a image model first and building, building these two models. The data you need is a hundred percent synthetic pair of language and image or language to video because on the, on the internet, actually the, the videos Don't naturally associate this text.”

“By training these upcycled models, you can achieve better accuracy than simply training the dense model further for the same number of flops.”

“we actually found the mix-throughs approach didn't work as well as expected, because the, the original model, the original switch transformer from Google uses a softmax and topk for a reason. And, uh, because of upcycling, if you switch to topk, then softmax, it actually hurts some performance.”

“There, once I built the Cosmos one, I realized as this thing also has a scaling law similar to language model.”

“When I look at like training models, I don't, uh, so actually the, the top important thing is like how many, uh, how many iterations can you do like per, per day? Uh, and the, the more iteration can you do, you can, you can train the model much faster. So if you have a very strong infra and you, you have a lot of…”

“Compute might become a bottleneck again, because previously, like if you want to train a new model, say you want to generate new synthetic data and then, or write a new algorithm, it might take a few weeks. And during that period of time, you don't, you might not have experiments to run. And now you, you can build that…”

“For the generative model training, there's also really like a small percentage of unlabeled data. So, so the model is instructed to generate a video without any text instruction. That, that can also help the model generalize.”

“if you're trying, if you can technically, theoretically train image or video models on pure pixels, but the, the problem is that the, it's, it's a lot of tokens. So like one, one image, like, uh, it's a thousand by a thousand is like one million tokens, one million pixels. It's impossible. To train transformer on that.”

“So now the training, training of the diffusion transformer, you already generated models use diffusion transformers. It is actually quite standard. It's, it's very similar to how you train a language transformer models. It's not that much difference. It's just the tokens, the, the visual tokens in visual tokens out.…”

“After you train such model, such image model, the reason it's a, it's a foundation for video models is that image, image models are Cheaper to train and they have much denser connection between language and text. So, sorry, language and images. For example, you, you train a billion, you train on a billion images and…”

“That being said, the benefit of the frame per frame compression, we might come back to this later, is, uh, real timeliness and interactivity. Because if you, if you strain the output of the model, uh, frame by frame, you can As a model can respond to any user request immediately. So if you have like a temporal four,…”

“I guess the, from the modeling perspective, the strong model, the teacher model is trying to model The image and videos of entire internet. And that distribution is extremely complex. As a step distilled model is just trying to learn from the teacher. The teacher is, is a model and the size is fixed as a distribution…”

“So word model is like real time, interactive, long horizon videos.”

“It doesn't need to have a very long context, but it's, I feel like it's an intermediate solution. It's cheating. Yeah, the model should Be able to like selectively know, like where, where should I draw references?”

“if you think about some limitation, for example, the current data, like how, how fast can we acquire the videos? And if, if you think about training the models, like, uh, what's the iteration speed? For training a model end-to-end and how, how would adding more GPUs accelerate that timeline? And maybe if you need human…”

“most of the people were actually not very good at prompting. Actually, language models have a better sense of how to prompt AI models. AI models know AI models better.”

“I think the enterprise will have much more budget for video models because the agents are inherently more expensive than, than the other video models themselves because they do this iterative process. They, they generate many, many variations.”

“From our experience, the heuristic engineering also have the models get absorbed into the models themselves.”

“a lot of, a lot of pre-training experiments show that Token dropping is very efficient, and it doesn't impact performance, but in some of the, like, the downstream fine-tuning, people realize drop-less is better.”

“We found, 64 experts is kind of like the sweet spot. If you increase the number of experts beyond 64, it provides diminishing return.”

“So people actually have tried that, but the, the main challenge is the latent space for the MP four tokens are not, we're not very comprehensible for the models. It's extremely hard to train on that.”

“The difference is if you compress the temporal dimension, you, you get a much higher compression rate. Because there is temporal redundancy between frames, because this frame and the last frame, likely they are mostly similar. So there's only some small difference. Uh, for example, like, uh, I think in one, 2.1, they,…”

“You use a VAE to compress the videos and you also need to store, typically you need to store those continuous feature on also in your storage. That's also comparable size with the videos themselves.”

“So one important thing is like the alignment. So the model, the model has to know like the video and audio, the, uh, it has to have a time based alignment, like at which time step the video and the audio token correspond to each other. We actually don't have these kind of alignment for, for, for most of the other…”