code

“Code also is a big piece of improving reasoning. So yeah generated code is not That much worse than, like, regular human written code. You might even say it could be better in a lot of ways.”

“Code is the most explicit example of reasoning data on the internet. Yeah. And it's not only structured, it's actually very explicit, which is nice. You know, it says this variable means this and then it uses this variable and then the function does this. Like…”

“Sometimes I say that like the LLMs aren't [4137] Great at planning yet. [4138] So we can help them plan by telling them how to plan and code. [4140] Cause that's very explicit and that's a good way of communicating how they should plan and stuff like that.”

“At least in my experience, rag from language to code doesn't work super well.”

“Capability in code is really core to performance on any LLM and like loosely, the better any LLM is at code, the better it is at any downstream task, even that's like writing poetry.”

“Code is one of the very few things, especially at the time that you could actually validate. And so you could have that agentic loop where the LM is generating the output and you're actually, you know, verifying in ground truth, the quality of that output.”

“For math, it's a bit harder to over optimize, I think. Unless you have tools and the model learns to search and cheat instead of learning math, which I'm sure somebody could see that out in the world, which is like, oh, I'll just find the, you're training. It'…”

“And I think of knowledge work as being on a similar sort of exponential as code has been on, but just time shifted, right?”

“But no matter what, there's going to be 10 to a hundred times more code. So I think you can be very long engineering right now as just a, you know, purely on the dimension of software is going to become increasingly more important. Once agents are, you know, t…”

“But it's a lot easier to change with code, as opposed to a model that is Learned primarily on data of real world and virtual worlds that are, I guess, like for example, junior, like there's actually trained on a lot of real world data and a lot of virtual gami…”