The Wisdom Wall

“the existing paper people are writing about multi-turn RL are not actually incorporating this, and it kind of, like, breaks all the math.”

“you can have a model trained on code, and a model trained on math, and a model trained on Spanish, and you can literally average the weights, and it works.”

“The updates made to model weights are orthogonal enough for specialized tasks that this is actually like totally fine. Things are nice and linear in most cases, things are nice and orthogonal, and you can get away with a lot of, uh, async, uh, updates to models that are then merged even without full communication.”

“a lot of the kind of headline anthropic like safety results, especially related to reward hacking and kind of deviation and alignment faking, Are all things to me that seem like a rock and a hard play situation where the model has two objectives it's given that are conflicting with each other, and it has to pick one.”

“I think being an eval company puts you in a really hard spot. Some people are talking about this on Twitter, like just that to be an ed-all company, you kind of have to sell to the labs, but selling to the labs doesn't really, like kind of wrecks the revals.”

“So it, like, feels like people are moving in the direction of model-based rewards, where you, either LLM is a judge where the judge sees the correct answer, or it has questions it's supposed to verify as properties of the response, just because that's much more flexible than, like, trying to write these little parsers.”

“where I think for a while some people were, like, speculating, like, oh, what if you have the model, like, generate images in its chain of thought reasoning where everything is, like, uh, end-to-end multimodal input and output, and it seems like you don't really need to do that because for a lot of the, like,…”

“and the most, one of the most promising ways, I think, towards doing this is having the reward models also be able to answer harder questions by themselves being reasoning models.”

“the thing that's going to make the next wave of stuff be powerful is just, like, everyone wants better agents. Everyone wants models that can, like, go off and do stuff. And, like, reasoning was kind of, like, a precursor to that a little bit.”

“So it seems like artificially truncating the thought is actually like fine. Like the model can, even if like it got cut off mid-sentence with an injected like think token, these are smart enough models that they can kind of finish with the best that they got from that point.”

“the base model in general of LLM is not artificially constrained in any way. Like, with the right prompt, it'll do whatever up to its intelligence limit.”

“if you set these models up to use tools, They just won't. Like if you say, hey, here's a question. You have access to these tools. Do as many rounds of tool calling as you want, and then submit your answer. They'll just submit their answer because they like are, especially for like small models, like they aren't…”

“if you want models to use thinking tokens, you kind of have to, like, incentivize that. You have to either do a little bit of, like, SFT warmup, or you have to, like, Reward them for doing it. Otherwise, they will not follow it a hundred percent of the time on format alone”

“if you start rewarding them for, like, tool use, They will use the tool, but they don't really want to, like, have to, they want to, like, be very safe with it... They would like do silly versions of tool use where they aren't actually using the tool to assist in their reasoning. They're using it to get the reward.”

“Um, and that, if you can crack that, uh, scaling direction of, like, pushing the boundary of how long these models can go out and do these things for, that is a plausible path towards things that become marvelous.”

“you probably want some kind of intermediate verification where you're not just waiting for the final answer at the end, But you want something like turn-level reward, potentially, where you want to be able to ensure that the model is getting credit for the, the moves it makes along the way, because it's no longer just…”

“you also ideally want to, whether you're doing this centralized or decentralized, move in the direction of, like, everything being async and overlaps, um, because that is, like, otherwise you have these, uh, inefficiency bubbles that, like, pop up all over your, uh, compute structure, your brain structure, and you…”

“Using more tool calls, searching the web more, gives you a nice scaling curve where you get better answers by putting in more effort, by, like, spending more time browsing the internet, essentially.”