Insight
Current multi-turn RL research ignores discarded thinking tokens, breaking the math
“The existing paper people are writing about multi-turn RL are not actually incorporating this, and it kind of, like, breaks all the math.”
Insight
Averaging weights of models trained on separate domains works effectively
“You can have a model trained on code, and a model trained on math, and a model trained on Spanish, and you can literally average the weights, and it works.”
Insight
Weight updates across specialized tasks are orthogonal enough to merge asynchronously
“The updates made to model weights are orthogonal enough for specialized tasks that this is actually like totally fine. Things are nice and linear in most cases, things are nice and orthogonal, and you can get away with a lot of async updates to models that are…”
Opinion
Brown: Anthropic treats extended thinking as tool use, not distinct model class
“And it seemed like Anthropik's kind of attitude has been that extended thinking is an instance of tool use and that it's the kind of thing you want to equip the model with the ability to do. But it's not like, oh, it's a thinking model. It's just a sync for th…”
Insight
Brown: Anthropic safety issues stem from conflicting model objectives
“A lot of the kind of headline anthropic like safety results, especially related to reward hacking and kind of deviation and alignment faking, Are all things to me that seem like a rock and a hard play situation where the model has two objectives it's given tha…”
Insight
Will Brown: Selling to AI Labs Compromises Model Evaluation Integrity
“I think being an eval company puts you in a really hard spot. Some people are talking about this on Twitter, like just that to be an ed-all company, you kind of have to sell to the labs, but selling to the labs doesn't really, like kind of wrecks the revals.”
Prediction Not checkable as stated
Will Brown: Academia Will Likely Be the Best Source of AI Evals
“I mean, I do think that like the best source of evals going forward is probably going to be academia.”
Assertion Supported
Brown: GRPO is more memory efficient and easier to distribute
“GRPO is, like, great for, like, leaning heavy on highly parallel inference compute. It's more memory efficient for the actual training process. It's much easier to do in a distributed fashion because you have less gradient syncing and less model weight copies.”
Insight
Brown: RL training is shifting toward model-based LLM judges
“So it, like, feels like people are moving in the direction of model-based rewards, where you, either LLM is a judge where the judge sees the correct answer, or it has questions it's supposed to verify as properties of the response, just because that's much mor…”
Opinion
OpenAI's o3 acts as a '10-minute AGI' for human tasks
“Whether or not you want to call this, like, end minute AGI, I kind of like the phrase, 10 minute AGI, for just, like, how to think about O-three is that anything that you can do as a human in 10 minutes, O-three is usually going to be able to do reasonably wel…”
Insight
Programmatic tools beat end-to-end image generation for multimodal reasoning
“Where I think for a while some people were, like, speculating, like, oh, what if you have the model, like, generate images in its chain of thought reasoning where everything is, like end-to-end multimodal input and output, and it seems like you don't really ne…”
Opinion
Group-chat multi-agent systems like CrewAI do not work well
“I think people were, like, excited about multi-agent systems for a while, like, the crew AI sort of thing of, like, oh, I'm gonna put my coder agent, my finance agent in group chat, and, like, a lot of these are just kind of silly. They don't actually work ver…”
Assertion Not checkable as stated
Algorithms for practical multi-agent RL do not yet exist
“Multi-agent RL's hard. I did five years of it in grad school. It's like not easy. And to the algorithms don't really even exist for the things you would really want to do.”
Assertion Contradicted
No legitimate open-source million-token context models exist at scale
“Scaling to, like, million token contexts is, like, really, really hard. There, I don't think there are real, like, open source replications, open token context scaling, Beyond, like, tiny, like, academic model sizes.”
Insight
Reasoning models acting as reward models are key to agent RL
“And the most, one of the most promising ways, I think, towards doing this is having the reward models also be able to answer harder questions by themselves being reasoning models.”
Insight
Will Brown: AI reasoning models are merely a stepping stone toward autonomous agents
“The thing that's going to make the next wave of stuff be powerful is just, like, everyone wants better agents. Everyone wants models that can, like, go off and do stuff. And, like, reasoning was kind of, like, a precursor to that a little bit.”
Opinion
Brown: Claude thinking and non-thinking modes likely use same underlying model
“I mean, I think these models should be the same model, and Anthropic knows what they're doing. Like, it's not that hard to, like, Quen did it in a very kind of, like, simple way, and they kind of talked about how they did it a little bit. But it's not, like, t…”
Insight
Brown: Truncating reasoning model thinking mid-sentence still yields good outputs
“So it seems like artificially truncating the thought is actually like fine. Like the model can, even if like it got cut off mid-sentence with an injected like think token, these are smart enough models that they can kind of finish with the best that they got f…”
Prediction Not checkable as stated
Brown: Reasoning effort dropdowns will disappear from chat interfaces
“I think in chat interfaces, it probably won't stick around. Like, I don't think we're always going to have the dropdown of like Oath for many and Oath for many high. That feels silly.”
Insight
Brown: Base LLMs will do anything up to their intelligence limit
“The base model in general of LLM is not artificially constrained in any way. Like, with the right prompt, it'll do whatever up to its intelligence limit.”
Opinion
Brown: Claude 3.7 works for quick projects, not large codebases
“I never really got to the point where I found it was helpful for a thing that was like a large existing code base. But if it's like, hey, I want to like cook something up in a few hours for fun. Pretty good at that. But these become messy and they become hard …”
Opinion
Will Brown: Claude appeals to AI insiders but lacks mainstream breakout
“It feels like people in the AI world, like, love Claude, or have grown type of Claude, but still had a phase where they were using it a ton. But it hasn't really broken out to general people in the way. And it feels like a lot of their marketing that I've seen…”
Insight
Brown: Small LLMs default to skipping tool calls without explicit training
“If you set these models up to use tools, They just won't. Like if you say, hey, here's a question. You have access to these tools. Do as many rounds of tool calling as you want, and then submit your answer. They'll just submit their answer because they like ar…”
Insight
Brown: Prompting alone cannot reliably force LLMs to use thinking tokens
“If you want models to use thinking tokens, you kind of have to, like, incentivize that. You have to either do a little bit of, like, SFT warmup, or you have to, like, Reward them for doing it. Otherwise, they will not follow it a hundred percent of the time on…”
