The Ledger, every show

“So I think as a inference cost come down, we are going to have generative UI for everything.”

“Video agents, mostly language models, they'll call these generative model, either it's a separate model or a diffusion head or whatever as tool. So this model can iteratively Refine the results or even like you generate longer content through a very long trend…”

“I guess by the end of this year is this is going to be a big hit. So the inflection point will be there and the videos generated by video agents can get to like production great quality. So it can be presented and it can be distributed in, in ads.”

“There were no, no infra, no data, and no model. And it just a few engineers, we built it in three months and released the first model, Grok Imagine,”

“So if you train your neural OS or neural computer on the standard screen recordings on the entire internet, the model can imagine completely new interface to interact with the computer.”

“So, so it's like just storing, storing the network, those costs, it's just I guess it would be a few millions per month to just storing everything, not to mention the GPU costs.”

“So the Glock Imagine video extension, it has historical context of all of the previous generated videos. It can it has a context of who is speaking and what objects have appeared and everything having that to generate the next video.”

“Once these models can use computers and understand the future state of computer extremely well, the robots might be Might be one of the tools a very powerful AI can use. So the powerful AI might just be able to control the physical embodiment naturally.”

“So Grok Imagine, there were .9, I believe it's is a first first audio video trends model deployed at a large scale.”

“I think the final state will be, for example, like a video version of Playbook where you can interact with a neural computer. You move your mouse and you click on the generative interface. And it will reply to you through, through pixels generally in real time…”

“But humans' contacts can, like, attention can work because we can dynamically pull in contacts from different places. The same mechanism I think it's going to happen for RLMs and video models.”

“He also worked very closely with people like people imagine online, like he, he's very hands-on.”

“As a limitation is like the technology is, as a paper, Was out there and people can reverse engineer that how to get rid of it. And it's, I think even as it advance, it's still, still possible to reverse engineer it.”

“So in, in Asian, in Gorky Imagine agent mode, you can still go in there and do, do stuff by yourself. Gradually, as the model capability increase, it will be able to do everything fully automated.”

“I think one thing pretty, pretty interesting. I think might be happening soon is the language models will be like context aware and manage its own context.”

“The Megatron MOEs was the first It was the first framework open source to be able to train these MOEs at very large scales, like a hundred billion parameters to even trillion parameters efficiently at like 40% MFU.”

“Let's also look at the implementation of Mixtro eight by seven on Hagen-Phys transformer. You will soon notice the, in the expert operation there, You would iterate over all of the experts and compute each of the gem operations one by one. We found that this i…”

“Unfortunately, people didn't find, like, a significant interpretability inside these experts. Say, one expert focus on math, the other focus on literature. I think the problem is that neural network hidden states are already very entangled. So, when hidden sta…”

“On other scaling experiments, we tried on 15 B models upcycling and applied on one trillion tokens and achieved roughly about five percent improvement in terms of the validation loss and four percent improvement on MMLU.”

“One thing I say, like, thanks to my experience at NVIDIA, because first time when we were building Cosmos together, we built it for about a year.”

“Yeah, for example, like in Cosmos, I think just five seconds of video is like a 50, 50 K or a 60 K number of tokens. So like, if you do 50 seconds as a 500 K tokens, if you do longer than that, easily explode.”