The Exchanges

Q in video compression where basically frame by frame, there's not that much difference. So actually you don't have to regenerate or resave the whole frame, right? Um, I think MP four compression or something else like that. Is it tempting to use that? Or as far as I can tell, everyone just treats it as, no, we will just generate every frame. Is that roughly the state of the art?

A There are a few different approaches. Let's say first, like you, you want to just directly use MP four compression and you use that as the tokens for The transformers to train, right? So people actually have tried that, but the, the main challenge is the latent space for the MP four tokens are not, we're not very comprehensible for the models. It's extremely hard to train on that. And there's a. So that's why they created VAEs, which creates more continuous latent space. So the models can understand that latent space and learn from it much easier. Even within the VAEs, there are different difficulties of the latent space. So you, you can imagine something that the simplest, the most naive VAE is like you, you have an image and you just shuffle all of the images into a Into a vector. So you don't need to train any of these, right? But that latent space is extremely hard for models to train on top of. So that, that's why there's some debate on like, how do you compress the, the tokens? So, so you mentioned like you can compress frame by frame. Also you can compress, uh, the temporal dimension. Yes. The difference is if you compress the temporal dimension, you, you get a much higher compression rate. Because there is temporal redundancy between frames, because this frame and the last frame, likely they are mostly similar. So there's only some small difference. Uh, for example, lik…

AI assessment note: “people actually have tried that, but the, the main challenge is the latent space”

Q I also wonder, have you, do you try using LLMs to look for bugs? I don't know.

A I remember at that time it was mid 25. So it's the, the coding model wasn't quite there yet. I remember, I remember like December, 25, it was extremely good. Yeah. I've been, I've been using it At that time, it's, it's helpful. Uh, sometimes it, it produce codes that are kind of difficult to maintain, even though like the first time it built something extremely fast, but it gave the, like a spaghetti code thousands of lines that I couldn't maintain and the OM itself couldn't figure out what's, what's wrong and how to improve on top of it. But now I find it much, much, much better. Yeah. I want to bring up another point here is like now coding models are much more efficient and can help, help us implement stuff much faster. Compute might become a bottleneck again, because previously, like if you want to train a new model, say you want to generate new synthetic data and then, or write a new algorithm, it might take a few weeks. And during that period of time, you don't, you might not have experiments to run. And now you, you can build that thing within a few hours, then you can immediately train a model. Now you have to have enough compute to, to try all of the ideas. So compute might be the bottleneck of iterating speed again.

AI assessment note: “I've been using it At that time, it's, it's helpful.”