The Ledger, every show

“Github Copilot or Cursor, which are incredible products. We use them internally. We're very happy with them. But again, this is, these are products where the engineer is driving most of the work. Like, even in agent mode, really the engineer is, like, driving …”

“So it's not just having long context, it's whether your model truly understands what's inside the context, and needle in the haystack tests are pretty crude and not very effective way of testing this kind of capability.”

“I bet would be on long context understanding and improving the attention mechanism over the long context.”

“Autonomous coding benchmarks, let's say, like Sweetbench, are useful. I'm not going to discount them. They are useful. But let's say, you know, 90% on Sweetbench could still mean something that just falls over flat within a customer setting.”

“I'm sure that other companies will converge on it as well.”

“And a lot of the project was in these big projects was not even on training the models. It was figuring out how the agents should actually interface with the environment that you're training it in.”

“When you look at what an engineer does in an organization, 80% of their time they're spending trying to comprehend complex systems and collaborating with teammates.”

“This is I think the least spoken about part of what frontier labs do, but Possibly the most important, which is figuring out how they evaluate, like what makes Claude magically feel better at code than you know, another model out there. They did something righ…”

“When you look at the model card for, let's say, the O-one paper that came out, I think, last year. If you look at the distribution of what most people worked on, on that paper, it was evals.”

“If you're looking at a file, and there's like a specific thing in that file, and you're just trying to get a quick answer to it, you don't really need the hammer of like a deep research like experience. You don't need to wait, you know, like tens of seconds or…”

“Where if you want to change the agents, the team wide memory, then it probably is going to look something like a pull request where the person who really understands that system Approves or, you know, edits it or something like this. I don't think it's going t…”

“When you look at how fast like Anthropics revenue is growing I think, right, they're kind of in this spot where it's like a massive revenue generating business that's growing at an unprecedented rate.”

“The RL methods we have today are quite bad, I would say, exploration and credit assignment. Like they, they're sort of just like the fundamental algorithms are take the things that work and make them happen more frequently, and the things that don't work and h…”

“Code forces and other competitive coding environments. The models are almost best in the world, and within the year will probably be just the best in the world.”

“The challenge is that if we, if you think that language model rewards are hackable vision language model rewards or, you know, like other sensory signal rewards are infinitely more hackable.”

“It can potentially work otherwise, but practically it only works when the agent has interacted with a reward, right? It's received a positive reward for what it's done. Maybe one out of 10 times, one out of 50 times, but if it's getting zero reward, then you d…”

“I think I think there are going to be a lot of move 37”

“Deep seeks recent open sourcing of their various code components that they use to train that model, which I think outside of the big labs was not really well known to, right, it was not really well known how to write kind of a kernel that's optimized for this …”

“As a company gets larger, the, like, engineer goes from spending most of their time on, you know, working on the features that matter, and the kind of more, the more kind of value-driven work at a startup, To a very large company where you have giant code base…”

“I joined a small project at the time that you know, was tens of people. And that project became Gemini one and 1.5, and then obviously two and so forth. And I joined with my co-founder, my co-founder, Yannis was leading the reinforcement learning team, the RLE…”