Base Model
topic on 5 shows · 16 statements across 16 episodes
More or Less 
Latent Space 
No Priors 
the MAD Podcast 
the a16z Podcast
16 statements about Base Model, every show
Training speculative decoding models requires hidden states from the base model
“Now, with speculators today, you need to train the speculator using the base model itself, because you're getting hidden states out of the model from running inference on these specific prompts, and that is the training data you use to create the speculator.”
Kant: Base model pre-training is required to unlock major capabilities
“You can't fine tune your way to success, right? Major capabilities emerge from training a base model made accurate and useful during fine tuning.”
Fredrikson: Prompt engineering cannot reliably enforce AI agent security policies
“Oftentimes people will try and prompt their way around it, like adjust the system prompt or like engineer the agent in a way where you're interjecting all the time and reminding it of what the original Goal and objective was, and that'll get you a little bit o…”
Updating mobile base models breaks per-app LoRAs, creating severe maintenance hurdles
“From a developer point of view, I think it will be very tricky because one, you don't want to have 20 different base models in the phone of the users. The battery will just die. You also don't want to have to update 20 LoRa every time you update the base model…”
Bachman: Power Retention models match original base model performance
“They'll come out with a nice shiny new, a power retention architecture that has the same performance on whatever data set they want as the original base model did.”
Fortuna: Base LLMs unsafely infer unconfirmed diagnoses from patient symptoms
“Patient, you know, they start to make these medical inferences. They're so smart, but they start to, like, infer things that the doctor didn't actually explicitly say. You know, for instance, a patient will say, like, I'm feeling sad and stressed out. Difficul…”
Laskin: Base language models before alignment are useless stochastic parrots
“If you play with one of these base models before they're aligned, they're really useless. They're, they are, they feel like stochastic parrots. They don't follow instructions.”
Brown: Base LLMs will do anything up to their intelligence limit
“The base model in general of LLM is not artificially constrained in any way. Like, with the right prompt, it'll do whatever up to its intelligence limit.”
Prompt routing scales better than model routing as base models improve
“For us, we've actually like figured that the like unit, the like The return on it was not actually that beneficial. What makes sense for us instead is, you know, like we use base models for like the final, like response. What makes sense, what makes more sense…”
Mensch: General-purpose base AI models will eventually become open source
“There are general purpose models like base models, compression of the web. That are eventually going to be open source and that can serve as the right basis for constructing specialized systems.”
Dohmke: Specialized code models were replaced by general base models
“And then we got this model that then eventually became Codex which was this code-specific, you know, version of the model. And today that, you know No longer really exists, right? Like today, everybody sits on top of one of these more powerful base models.”
Sam Lessin: AI Applications Will Primarily Be Scaled Persona Characters
“You're going to end up with like, you got to just think about like the base model and then what's being done on top of them is like, they're all just characters. Different people at different wealth levels and power levels are going to, like, invent those char…”
Sam Lessin: AI base models will be competed down to zero profit
“You're not going to make money on base models. Right. They're going to be competed to zero.”
Horowitz: AI economic value will accrue to tools, not base models
“The actual layer where the value is gonna crew is gonna be like tools, orchestration, that kind of thing, because you can just plug in whatever the best model is at the time. Whereas the models are gonna be competing, you know, in a death battle with each othe…”
Sam Lessin: AI Base Models Will Either Be Worthless or Have Two to Three Winners
“There's no stable configuration for the base models where there's like a hundred of them. There's either going to be Zero that make any money. So they all are worthless, right? Cause it like, cause it is everywhere. And like, for what it's worth, that's been m…”
Ratner: Base models will open-source while production uses specialized models
“There are going to be these base models. They're probably going to be increasingly open source. And then the reality of what actually ships in production is going to be a whole kind of family tree of smaller specialized models that are tuned and adapted via da…”