Model Weights
topic on 6 shows · 7 statements across 7 episodes
Latent Space 
Invest Like the Best 
the MAD Podcast 
How I Built This 
the a16z Podcast 
TBPN
7 statements about Model Weights, every show
Movva: KV cache frequently exceeds model weight size
“You have to store a representation for every token that we sent through the language model. And it frequently Gets to be larger than the weights of the model themselves.”
Trojanowski: Basis Prioritizes Agent Orchestration Over Model Weight Updates
“Today we don't go directly into the weights. And primarily the reason we don't do that is because A lot of the advancements the models are having when it comes to orchestrating themselves yield far more performance gains than benefits you would have of like up…”
Rogers: Disclosing AI model weights compromises Western competitive edge
“If you force companies to disclose aspects of the AI that let foreign adversaries, for example, reverse engineer the model weights, you're really compromising the American and the Western competitive edge.”
Physical chips, not model weights, are the refined uranium of AI
“Maybe the chips are the refined uranium more than the actual weights, and the weights are merely one piece of the puzzle.”
Eskildsen: Model weights compress reasoning, not all world knowledge
“We can take all of the world's knowledge, all of the exabytes and exabytes of data that there is, and we can use those tokens to train a model, but we can't compress all of that into a few terabytes of weights, right? We can compress into a few terabytes of we…”
Morris: Weight deltas can reconstruct a competitor's proprietary fine-tuning dataset
“There's some tricks to it, but it's basically just like gradient based selection based on this weight difference. And it seems to be okay. Like it can get us pretty good training data. So I guess if you actually wanted to use this, it would be like your compet…”
Altman: Open-sourcing AI model weights is an irreversible one-way door
“If we just publish the weights of a model on the internet, and then we realize, like there's actually a safety issue here, we can't take that back. It's done. It's like a one-way door. We also cannot put any usage restrictions on it after we open source it.”