The Ledger, every show

“Get used to it because this is as normal as it's going to be. It's going to be much weirder very soon.”

“Some of the things that are dramatically better are, for example, in coding, it is able to not do it sort of off target mutations or over eagerness or reward hacking.”

“In coding in particular, we've seen some customers using it for many, many hours unattended and doing giant refactors on its own.”

“If you give Opus a tool, which is Sonnet, It can use that tool effectively as a sub-agent. And we do this a lot in our agentic coding harness called Cloud Code. So if you ask it to like look through the code base for blah, blah, blah, then it will Delegate out…”

“But at the same time, as a user, you don't want to have to decide yourself, does this merit more dollars or less dollars? do I need the intelligence? And so I think having like a routing layer would make a lot of sense.”

“We haven't started testing it rigorously yet. I mean, we have had our models take our interviews and they're extremely good. So I don't think that would tell us, but yeah, interviews are only a poor approximation of real shot performance unfortunately.”

“And then on the data side, RL environments are really important these days, but constructing those environments Has traditionally been expensive. Models are pretty good at writing environments, so it's another area where you can sort of recursively self-improv…”

“As we've trained the models more and scaled up a lot, it's become harder to find humans with enough expertise to meaningfully contribute to these feedback comparisons. So for example, for coding, somebody who isn't already an expert software engineer would pro…”

“Like for example, we're working with Novo Nordisk and it used to take them Like, 12 weeks or something to write a report on cancer patient, what kind of treatment they should get. And now it takes like 10 minutes to get the report, and then they can start doin…”

“Take a portfolio approach, try some less risky use cases, like where knowledge extraction or summarization might be involved. And maybe some more risky use cases like tool use, where it's using that company's tooling, function calling.”

“In our case, we only have two models and they're differentiated by like cost performance Pareto frontier.”

“One large bank that we were talking to, they said, they came to us and they said, we've, Talk to everybody in our company, and we have 500 different use cases that we want to apply large language models to.”