Anthropic

“And I do see that's going to shift the balance of it. More and more folks are going to be using let's say derivatives of Lama two. More folks are going to Fine-tune and serve their own model instead of calling an API.”

“There's a whole lot of technical debt everywhere, you know, nobody's really figured this stuff out because everybody's been so busy building what we know works as quickly as possible. So, yeah, I think there's a huge amount of opportunity to, you know, I think…”

“One of the big bottlenecks is how much power and how many chips you can get into a single data center. So, like, A, Google and OpenAI and Anthropic are working on this, right?”

“The constitutional AI and the super alignment is, like, very conceptually linked. It's like a group of people that has, like, a very similar intellectual upbringing, and they work together for a long time, like, coming to the same conclusions in different ways…”

“Anthropic and ChatGPT and Bard almost surely have a classifier after, which is like, is this text good? Is this text bad?”

“If you look at Claude's constitution, like, that doesn't mean the model believes these things. It's just trying Trained and to prioritize these things.”

“Fun fact is that these loss functions Look different and anthropic in opening eyes papers, but they're just literally just log transform. So if you start like expantiating both sides and taking the log of both sides, you'll like converge on one of the two, the…”

“At the start of twenty-twenty-three, Anthropik kind of launched their constitutional AI paper and within a few days, I think four days, he had basically implemented that in production, and then we had it in-app, like, a week or so after that, and he has since …”

“Overall, I'm like super happy with the anthropic models compared to the OpenAM models. Like, Sonnet is very cost effective. Haiku is, in function calling, it's actually better.”

“Like sonnet will still create things that kind of like floor you sometimes. Opus can just handle much more complexity. I'd say it is the big heuristic there.”

“With Claude, we notice the system prompt is written in third person. It's written in third person. It's written as, the assistant is X, Y, Z.”

“You can point this out to Claude that a lot of the assumptions that it has in its behavior are actually inconsistent with the communicative goals that it has in this situation. It leads it to notice these inconsistencies and gives it more degrees of freedom.”

“Instead of having this entity, like GPT-IV, that's an assistant that just pops up in your face that you have to kind of, like, punch your way through and continue to have to deal with as a headache, instead, there's ways to kindly coax Claude into having the a…”

“Anthropic actually cheats at this right now. If you look at the system prompt in, in the cloud artifacts, I actually have a thinking section that is explicitly removed from the output, which is, I mean, they're still spending the tokens, but like that is befor…”

“They are partnered with AWS, and it's going to be in bedrock. As far as I know, I think that's true.”

“It's just not practical outside of OpenAI to run use cases at scale in a lot of cases. Like, you can do it, but it requires quite a bit of work. And Because OpenAI is so good at making their models so available, I think they get a lot of credit for the science…”

“We still see an overwhelming majority of customers using OpenAI, but almost everyone is using Anthropic for the, and Sonnet specifically for their side projects, whether it's You know, via cursor or prototypes or whatever.”

“They kind of innovated in an interesting way, which was never quite publicized, but it's that they have that kind of chain of thoughts step whenever you use a Clouds model or Sonnet model with function calling. That chain of service step doesn't exist when you…”

“What I understood of it is that there was a disagreement of the commercialization of that technology. I think the focal point of the disagreement was the fact that we started working on the API and wanted to make those models available through an API. Is that …”

“My philosophy about it is anything that can be done with an API must be done by an API or should be done by an API for a very long time.”

“With this blog post we released on SweetBench, we released the exact tools and the prompt that we gave the model to be able to do well.”

“I think with agent frameworks in general, they can certainly save you some like boilerplate, but I think there's actually this like downside of making agents too easy, where you end up very quickly, like building a much more complex system than you need. And s…”

“We did a few different experiments with like different ways to specify how to edit a file and string replace. Basically the model has to write out the existing version of the string and then a new version, and that just gets swapped in. We found that to be the…”

“The biggest limiting thing will start to become like, do people trust the output of these agents? And like, how do you trust the output of an agent that did five hours of work for you and is coming back with something? And if you can't find some way to trust t…”

“You know, we're not going to go and do lots more submissions to sweet bench and try to try to prompt engineer this and build a bigger system. We want people to like the ecosystem to do that on top of our models.”

“I think most broadly, not just for like new things that weren't possible before, but as a much lower friction way to implement tool use.”

“And I think like the smarter the models are, the less you need that kind of extra scaffolding.”

“You know, I think that might be useful for hobbyists and demos, but the ultimate end applications are going to be bespoke. And so we just want to make sure that the model's great at any tool that it uses”

“I would say actually we did more engineering of the tools than the overall prompt.”

“And the opening I spend at the beginning, at the end of last year in November of 23 was close to 90% of total volume. And today, less than a year later, it's closer to 60% of total volume.”

“We already have an API for GitHub. So why do we need an MCP for GitHub, right? You know, like GitHub has an API. The GitHub API is evolving. We can look up the GitHub API documentation. So it seems like kind of duplicated a little bit. And also they have a set…”

“I still am under the impression that Claude is the best. The other closed models are, you know, not quite as good, and then the open models are a little bit behind that.”

“So, like, GPT doesn't have very good air recovery ability. And so, because of this, it will go into loops and do the same thing over and over and over again, whereas Claude does not do this.”

“Canvas has incorporated the diff mode that both Anthropic and OpenAI and Fireworks has now shipped that I think is going to be the norm for next year, that everyone Need some kind of diff mode code interpreter thing.”

“The other one would be Bolt. There's a straight quad wrapper. And again, another now they've announced twenty million ARR, which is another step up from our eight million that we put on the title.”

“And so I pick and OpenAI have both keys that model routing on APIs already. And so I think they'll launch them, especially at some point where you can sort of prioritize the three tradeoffs that are in model routing, cost, speed, intelligence.”

“I was working on something similar to, like, Canvas-y, but for Claude at that time, in, like, twenty-twenty-three, it was the same similar idea of, like, Claude workspace where a human and a Claude could have, like, a shared workspace which is like a document.”

“I think it was like the first, I think we were the first lab, like, Antarctica was the first lab to, like, run. Publish GPQA, like, numbers”

“I was a part of the post-training fine-tuning team. We only had, like, what, like, 10, 12 people involved”

“People said, like, Cloud II is, like, so much better at, like, writing and, like, has a certain personality, even though it was, like, unintentional at all. And we did not pay that much attention and didn't know even how to, like, productionize this property o…”

“I have been very bearish in computer use because they're slow. They're expensive. They're imprecise. Like the accuracy is horrible. Still, even with Anthropix new stuff, I'm really waiting to see what opening I might do to change my opinions.”

“Yeah, like I think like the first like 50,000 code of lines without any reviews at that time because there's no one. Yeah, it was like very small team. It was like six, seven team who we were called a deployment team.”

“OpenAI and Anthropik is different in terms of like more like maybe like product mindset. Maybe OpenAI is much more willing to take some of the product risks and explore different bets. And I think Anthropik is much more focused and they have, I think it's fine…”

“There's a case where a lot of people are going desktop first for, you know, consumer use. And I think Claude is doing a lot of this where I expect to see, you know, MCP is really oriented around the Claude desktop app for a reason, right? Like I think a lot of…”

“There's like at least thousands of dollars of tokens being consumed. So it's not a, it is not a cheap rollout.”

“The run that you saw that's, like, on the graph we put out alongside, like, in our research blog is, like, a single run that I have watched, like, get through At least surges Jim. And then it got a little past that. And the reason that that's where we stopped …”

“I really do think like this is just demonstrating like a thing that is going to make agents better with this model, you know, like This is a very fun way to see it, but, like, I think the thing is that it, like, has some ability to, like, course correct, updat…”

“Quad doesn't particularly understand, like, the middle of a Game Boy screen and a whole bunch of concepts like that, which means, like, you can prompt all around everywhere, but, like, this kind of, like, spatial awareness and where something is with respect t…”

“So in practice, this rollout ends up, like, at max, ending up around a 100,000 tokens, I think, is where it is, like, the longest message you ever send to the API on one of these turns, and it will fluctuate in, like, summarization, depending on the state of k…”

“Like, in my opinion, Claude is the best one when it comes to the way it formulates things.”

“Even GPT 4.5 is too expensive. Normally it's really gonna use it in, in any reasonable quantity. Like, you know, Claude 3.5 Opus, like if it does exist, still not like, you know, the thing that we actually use is Sonnet, right? So like, it's almost like a depl…”

“The thing that Claude's the worst at is understanding how to get from point A to point B. It's, like, really, really god-awful at hitting the buttons to go from point A to point B on a screen.”

“Building evals that actually test, like, time performance over, like, days of token sampling are quite hard. Like, it's actually really, really hard to build things that you can measure In a reasonable way. And one, this is one of them. Like, it's an eval that…”

“Anthropic actually did like a blinded study of like named characters versus unnamed characters in different settings, and Claude like actually does clearly prefer and is nicer to named characters, which is an interesting thing.”

“The entirety of Open Router that was not Anthropic or Google like, or Gemini or OpenAI or something. It was like, 10 H 100 nodes or something like that. It's just, like, not that much. It's like, not that many GPUs, actually, to service that entire demand.”

“Google would be a two to one crawl to referral ratio, so for every two pages, they will read, they will send you one visitor. He said OpenAI is 250 to one, so they'll read 250 of your pages and send you one person. And Anthropic was like 6000 to one. So they'l…”

“I used the same project to do this, to try to repeat the demo that I made for myself a month afterwards, and it wasn't anywhere as smart. So cloud three got dumber, but it looks like I like made up the demo or something, but no, like literally I just reran the…”

“There was a really early version of Cloud Code many, many months ago, and this one engineer at Anthropic, Jeremy, built a bot that looked through a particular feedback channel on Slack, and he hooked it up to code to have code automatically put up PRs. With ju…”

“The latest Sonnet three seven is, it's a very persistent model. It's like very, very motivated to accomplish the user's goal, but it sometimes takes the user's goal very literally, and so it doesn't always fulfill what, like, the implied parts of the request a…”

“We tried a bunch of different options for compacting, you know, like rewriting old tool calls and truncating old messages and not new messages. And then the end, we actually just did the simplest thing, which is ask Claude to summarize the, you know, the previ…”

“Originally we tried very, very early versions of Claude actually used RAG. So we like indexed the code base and I think we were just using Voyage.”

“And WhatMD, it's another example of this idea of, you know, do the simple thing first. We had all these crazy ideas about, like, memory architectures, and, you know, there's so much literature about this. There's so many different external products about this,…”

“Then we gave all the engineers and researchers at Anthropic access and pretty soon everyone was using it every day. And I remember we had this DAU chart for internal users and I was just watching it and it was vertical like for days”

“And there's some people at Anthropic that have been racking up like thousands of dollars a day with this kind of automation.”

“So we use Bunn to compile the code together.”

“Because Cloud Code is an MCP client and an MCP server.”

“And she's landing PRs to our console product. So it's not even just, like, building on quad code. It's building, like, across our product suite in our monorepo.”

“If we want some product that has like very broad a product market fit today, we would build, you know, a cursor or a Windsurf or something like this. Like these are awesome products that so many people use every day. I use them. That's not the product that we …”

“We're spending a lot of time building out the permission system, so Robert on our team is leading out this work. We think it's really important to give developers the control to say, hey, these are like the allowed permissions. Generally, this includes stuff l…”

“And it's funny with, when the model is so good, the simple thing usually works. You don't have to over-engineer it.”

“We plan to build in more native ways to handle this specific workflow.”

“It was, ah, Adr inspired Clyde, which inspired Cloud Code.”

“We think of it as like a Unix utility. Right. So it's like the same way that you would compose, you know, grep or cat or oh, cat. Or something like this. The same way you can compose code into workflows.”

“We have some experiments where Quad is doing code review internally. We're not super happy with the results yet, so it's not something that we want to open up quite yet.”

“And one thing we find is that people that are really good at prompting models, From whatever context, maybe they're not even technical, but they're just really good at prompting. They're really effective at using code. And if you're not very good at prompting,…”

“When I joined Anthropic, I was experimenting with different ways to use the model kind of in different places. And the way I was doing that was through the public API, the same API that everyone else has access to.”

“All the secret sauce, it's all in the model and this is the thinnest possible wrapper over the model. We literally could not build anything more minimal. This is the most minimal thing.”

“Probably near 80, I'd say.”

“The two that we're really trying to nail down are, one, decrease in cycle time. So how much faster are your features shipping because you're using these tools? So that might be something like the time between first commit and when your PR is merged. It's very …”

“Most of our research team actually uses quad code day to day, and so it's been a great way for them to be very hands-on and, like, experience the model failures, which makes it a lot easier for us to target these in model training and to actually provide bette…”

“And so in the UI, when you say, when you see a task that's actually like a sub-Claud, it's a sub-agent that does this. And usually when I do something hairy, I'll ask it to just investigate, you know, three times or five times or however many times in parallel…”

“Currently we're seeing costs around, like, six dollars per day per active user, and so it's, like, it does come out to a bit higher over the course of a month in Cursor but I don't think it's, like, out of band, and that's, like, roughly how we're thinking abo…”

“They established a Moore's law for time between human input, basically, and it's basically doubling every three to seven months is the idea. And Enthopic is currently doing super well on that benchmark. It's roughly about autonomous for 15 minutes at the 50th …”

“For quad code internally in the GitHub repo, we have this GitHub action that runs. And the GitHub action invokes quad code with a local slash command. And the slash command is lint. So it just runs a linter using quad. And it's a bunch of things that are prett…”

“You mentioned internal tools, and that's actually a really big use case that we're seeing emerge, because a lot of times if you're working on something operationally intensive, if you can spin up a internal dashboard for it, or like an operational tool where y…”

“We've rewritten it from scratch, yeah, probably every three weeks, four weeks or something, and it just like all the, it's like a ship of Theseus, right? Like every piece keeps getting swapped out, and just because quad is so good at writing its own code.”

“Yeah, we're kind of bordering on three right now.”

“Anecdotally for me, it's probably two X my productivity. So I'm just like, I'm an engineer that codes all day, every day. For me, it's probably two X. Yeah. I think there's some engineers at Anthropic where It's probably 10 X their productivity”

“Yeah, this is, it is, it's all chain of thought, actually, in quad code. So we don't use the think tool. Anytime that quad code does thinking, it's all a chain of thought.”

“And it seemed like Anthropik's kind of attitude has been that extended thinking is an instance of tool use and that it's the kind of thing you want to equip the model with the ability to do. But it's not like, oh, it's a thinking model. It's just a sync for th…”

“A lot of the kind of headline anthropic like safety results, especially related to reward hacking and kind of deviation and alignment faking, Are all things to me that seem like a rock and a hard play situation where the model has two objectives it's given tha…”

“I mean, I think these models should be the same model, and Anthropic knows what they're doing. Like, it's not that hard to, like, Quen did it in a very kind of, like, simple way, and they kind of talked about how they did it a little bit. But it's not, like, t…”

“It feels like people in the AI world, like, love Claude, or have grown type of Claude, but still had a phase where they were using it a ton. But it hasn't really broken out to general people in the way. And it feels like a lot of their marketing that I've seen…”

“And it does seem like Does sort of like inner representations have a higher connection to like the output logits for English logits. And so there's like some bias towards English at least in the model we studied here.”

“And two, this plan doesn't just control, like, what you're gonna rhyme with. It's also doing what's called like backwards planning, where it's like, well, because I need to finish with green, I'm not going to say illuminating the peaceful night, because then I…”

“The notebooks themselves They can all be run on Google Colab and all of the code, as far as we can tell, we've like tested on the notebooks, just like runs on Colab. And so that means that like, you don't need on a free tier to be clear, like you don't need li…”

“The way the circuit looks in Gemma, like a really small model is extremely similar to the way that it looks like a huge model, which that in itself is, I think like a pretty novel discovery. It's like, oh, you have these models that are like super different. Y…”

“When we had a team meeting, like it was a couple months ago, somebody on the team asked how many of the people on this team are here, at least in part because they like read one of these papers and thought like, wow, this is so compelling. Like this like makes…”

“Language models are next token predictors is like a fact. Like that is what they do. They are trained to predict the next token. However, that does not mean that they myopically only consider the next token When they choose the next token, you can work on brea…”

“I think for Interp in particular, there's like another thing that makes it easier to transition to, which is maybe two things. One, you can just do it without huge access to compute. Like, there are open source models. You can look at them. A lot of Interp pap…”

“There's been a lot of work in sort of like automated feature interpretability. And it's something that we've invested in and that like other labs have invested in. And I think basically the answer is we can definitely automate it and We're definitely going to …”

“These are just MLPs. So the model has both attention heads and multi-layer perceptions MLPs. We don't just do it. Like we completely ignore attention or like we don't try to decompose it at all. So there's some prompts where like all of the interesting stuff i…”

“So I think one example of this is like math where the model is like independently computing the like last digit and then the like order of magnitude and then kind of like combining them at the end or like hallucinations are also that where like, there's one si…”

“And then the release this week sort of lets anyone do it for a set of open source models. So notably maybe the most easy one here is like Gemma two to be. So you can sort of like think of some prompt and you kind of like can explain any like token that the mod…”

“We claim that this is like the Texas representation. Let's get another one and replace it. And we just change like that feature in the middle of the model and we change it to like California. And if you change it to California, sure enough, it says Sacramento.…”

“So I think there's like a sense in which right now the chain of thought is, is unfaithful, or at least you can't read the chain of thought and trust that that's how the model did it.”

“One of the reasons that we're really excited about this method is once you've built your like infrastructure, like to go from a prompt to like what happened is, you know, O of minutes.”

“And it turns out scaling it. I don't want to say it just worked because it was a lot of work. I don't mean to apply. There was an effort, but it worked. And now we're in the phase where it's like, oh, cool. These methods work on the models that we care about.”

“I bet, I don't know how much I bet a hundred bucks. So somebody can like, they would get a hundred bucks from me if they prove that I'm wrong, that this behavior for a model that does a drink fine tuning, it also does it post pre-training.”

“And even more recently, we released some code in partnership with the Anthropic Fellows program. It was mostly built by Anthropic Fellows that lets people play with the research basically.”

“Golden Gate Claude was like a pure, as far as I remember, at least, like, a pure, just like, Weird random thing where, like, somebody found it, initially went an internal demo of it, everybody thought it was hilarious, and then that's sort of how it came out. …”

“So like, I think that right now we have some pretty good solutions for like understanding what's in the residual stream, understanding what's, is it in MLPs? We don't have good solutions for like attention.”

“That means that, like, if that's true, then you can, like, set that feature to zero, or artificially set to a hundred, And you'll change model behavior. That's what we did when we did Golden Gate Claude, in which we found a feature that represents the directio…”

“The reason for publishing this is that we think interpretably is important. We think it's tractable, and we think more people should work on it. And so publishing it helps us like accomplish with these goals all these goals, which we think are just like crucia…”

“If you look inside the model, if you look at the middle of the model, which is the middle of this plot here, models share more features. They share more of these representations in the middle of the model, and bigger models share even more. And so the, like, t…”

“A team at Anthropic trained a model to have like weird hidden goals and then gave it to a bunch of other teams and said, Figure out what's wrong with it”

“ChatDBT isn't even the model of choice at this point. It's anthropic for us.”

“I think even hearing the cloud code folks on your podcast, like I would not be surprised if Anthropic like launched a thing that looks a little bit more like warp where it's like a harness for running a whole bunch of different cloud codes, but it's an actual …”

“Cursor I think is some very significant portion of their revenue.”

“So based off of a basic conversation, a single agent architecture for research is around four X, the number of tokens needed to achieve a research output. When you use multi-agent architectures, it's actually 15 X the number of tokens.”

“They initially started with five LLM as judges. So each one of these points had their own LLM as a judge. They tested the ability and accuracy of that LLM as judge collective to judge, and it actually didn't perform as well as one. So they replaced all of thos…”

“So the, in the blog post that Anthropik posted, they ran some tests and they noticed that the multi-agent structure outperforms the single agent structure by 80% based off a different variety of variables they measured.”

“When we released the leaderboard and just in a week that launched 3.7, And that went straight up, and nobody has beaten it so far.”

“I think the entire ecosystem will just converge around whatever they do. They just have such good distribution and they're, yeah.”

“Claude, which is important for, you know, not necessarily huge in terms of raw number of users, but the people who do use Claude tend to be like a very valuable audience, especially for some types of company.”

“We've had immense success getting people onto the platform, and we've been very fortunate to have the issue of being some of Anthropic and OpenAI's largest consumers.”

“The model spec is much more useful than a constitution because the constitution is like an intermediate training artifact that you give to the training algorithm in order to get the model that you want. It is not necessarily like what model did we, like we don…”

“Our friend Simon Willison wrote a post that Anthropic added Brave Search as one of the sub processor in their product.”

“I mean, when I saw the acquisition, I was like, oh, this is actually more intense now because the cursor team is going to move faster than the cloud code team. I think that's my feeling given they have to, there's more pressure on them and they're smaller.”

“I think it's pretty easy to, like, use more than 200 dollars worth, even by accident. So I would also lean towards that the Claude Max plans are a growth strategy, not like any long term pricing thing that can work, at least at the current, given the current s…”

“What would change things is if there's a day where either another LM lab or like, you know, an open source model drops that is competitive with Sonnet, maybe even better than Sonnet on that day, open code is going to be the only way to do this kind of thing. C…”

“We look at cloud code. We dump all their system prompts. We dump all their tool descriptions. We dump all their tool schemas. We re-implement the tools. And when you're using an anthropic model, we basically have the exact same implementation.”

“We know that they went there, they were there for, I think, about two weeks, and they came back.”

“And you know, one thing Chroma released this context rod paper recently about context utilization and the cloud models are actually the best at using kind of like longer context.”

“So we, right now essentially live on top of Sonnet IV, and we've tried a bunch of other models. We found that to work really, really well for us.”

“Language models, hosting language models is not a good business. At the time we thought, okay, we are going to be competing against OpenAI and Anthropic and all these labs. Turned, turned out that it was even worse because the killer application of language mo…”

“Anthropics multi-agent research is another nice example of this. They mentioned that the typical production agent, and this is probably referring to Cloud Code, could be other agents that they've produced, is like hundreds of tool calls.”

“If you use Cloud Code, you hit that Kind of, you know, you've hit 95% of the context window, and you're about to, and Cloud Code's about to perform compaction.”

“I like the take that apply multi-agents to problems that are easily parallelizable, that are read-only, for example, context gathering for deep research, and do, like, the final quote-unquote write, in this case report writing, at the end. I think this is tric…”

“Anthropic reported on this too. So their deep researcher just uses parallelized subagents for research collation, and they do the writing in one shot at the end.”

“Clock code doesn't do any indexing. It's just doing, quote unquote, agentic retrieval, just using simple tool calls, for example, using grep, to kind of poke around your files, no indexing whatsoever, and obviously works extremely well.”

“Anthropix kind of cloud models kind of had a regression, right? Because they changed to a new serving architecture.”

“If I give it this other custom-made MCP that we built internally, and our processes don't map to anything that OpenAI and Anthropic have seen or trained for, it won't be used, and you won't get good results.”

“I think we don't have any set evals. We don't. And this was controversial up until a week ago, I think, when I think Boris from Or two weeks ago from Anthropix that they don't have evals for the coding agent too. But we don't, and we haven't had them.”

“When cursors switch from Sonnet to GPT-V as like the default model that was like, you know, Two hundred million our revenue for Anthropic that kind of went away and like moved on to GPT-V.”

“I think we are one or two months away from a possible news cycle. That is the foundation model companies have spent billions of dollars in capex and hired like crazy. And now, you know, they're no longer the best in this realm and there's a huge stampede away …”

“Even internally we have found that having Cloud generate UI on demand for internal dashboards is useful, not just like an interesting demo.”

“And that thing's never gonna have an MCP around it. Like, it's just like, who knows if the company created is even around much less like ready to sort of expose their kind of underlying constructs as API. So I think you will need to be able to do both.”

“There's Cloud Code, which is directly built on top of the Cloud Agent SDK, and then there's external companies building on top of the Cloud Agent SDK, and I think we'll also offer ways in which if you want to run an agent off of that SDK, but have a lot of the…”

“There's, Cloud AI, and I think you'll see us start bringing that harness into Cloud AI more and more for things like document creation, advanced research, and so it'll power a lot more of the agentic workflows in there.”

“So this, you know, we had a customer and internally, we also got like a 30 hour plus kind of execution versus I think Opus four was seven hours.”

“This is like officially good. It's like better than Opus at this. It's like, It generated this, like, great split-screen stereoscopic thing, three-dimensional, like, thing.”

“We have more traffic on Sonnet 4.5 than we had on Sonnet four. So basically it's already eclipsed Sonnet four.”

“What was interesting about this model in particular is that it was really the first one where Product was upstream of research and downstream of research.”

“For some of the agentic part of the stack, we are shifting towards Anthropic because they're agentic and the tool calling, especially the unstacking part, you know, when you go down the wrong path and you build context that forces you to keep going down the wr…”

“Right when I joined OpenAI, like, this has actually been, I think, a passion project of John's. Like, he's been talking about doing something in this, like, in this shape for a while, which is, like, a truly, like, low-level research, like, fine-tuning library…”

“I think one of the really key moments for us was getting onto the Claude IV model card. Being one of two benchmarks that Dario actually mentioned while releasing the model.”

“OpenAI Anthropic, everyone else, they're all building models that are like maximally helpful. And in Actually, most cases in a corporate environment, you don't want that to be maximum. You don't want the model to be like helpful in every way possible.”

“If Anthropic ever found The activations for reasoning and could break down the different kinds of reasoning and turn, tune them properly. I think that's the thing that takes Anthropic to leapfrog OpenAI.”

“The high performance inference kernels that sort of drive a lot of, you know, anthropic and open AI and stuff, their models, those aren't open source. They're not going to be open source.”

“Now it's like basically kind of de facto one as the interop layer for all the labs and all the models.”

“I'm referring to the exact same article where I think their retention, one year retention on employees is the 80%, which in AI world is, is quite wild.”

“At that stage, to be very honest with you, at the stage that we invest in Anthropic now, like the only things that would really move the needle on the decision is here's the revenue, here's the margin, and here's the trajectory, and here's the other markets we…”

“It is far easier for Anthropic to try to go into one of the spaces of the apps than an app to try to go into the space of Anthropic, which makes me feel like one is more defensible than the other all else equal.”

“If you really want to go deep, I don't think you will ever dedicate the people to do it. And the last thing I'll say is you think about from an anthropic engineer's perspective, you joined a big AI lab to work on models, not to build Google drive connectors, r…”

“Look at the revenue of Anthropic and OpenAI right now. These are billion dollar revenue scale businesses. Glean is several hundred million dollar revenue scale business.”

“But the interesting about Anthropic is if you look at coding, that's probably never going to be the case. Like there's always an increasing frontier of how you good you could be at a task like that. And we're nowhere close to that frontier. So it's more possib…”

“In terms of enterprises, often what will happen is they'll buy up large chunks of long-term compute and dedicated instances, in which case you just don't churn, right? Like this is what you use.”

“Anthropic is the fastest growing software company of all time. I think I can say that fairly. I'm, I haven't been disproven yet.”

“Our business model is initially going to be an API, again, like the Anthropic API”

“I think he comes from a particular position of extreme unusual privilege, which is that he works at an AI lab where like people in the office next door are like writing the evals and are like training the model like every day in exactly that way.”

“I'm Vercel and I publish at Eval. That I want OpenAI and Anthropic to use to make sure when they ship the next model that they're better at the stuff that I care about.”

“We're working right now with Anthropic to basically do kind of like a building on the fly, small, kind of a key component of lambdas that will build the code to do the aggregation.”

“That whole thing ended with no open sourcing of data, but they did add a 30,000 or 20,000 dollar bounty, which I sort of sat myself out of, let the community go for it.”

“They don't have enough researchers to explore the entire latent space on their own. And so I think many hands make light work”

“They're hiring like a hundred plus people for cloud code in the next, I don't know, month. I mean, like they're going wild and that's just cloud code.”

“All three of those companies, Google, Anthropic, and OpenAI are unbelievably chaotic internally right now.”

“Like the commitment of Anthropic is the same, right? I'm still, We still have the same people I'm helping with the SDKs. We're still super committed in our products to MCP. I'm still the lead core maintainer. Nothing has actually changed.”

“We had a version of Goose that was still, you can go check the GitHub history. It was there a little bit before MCP came out, and we were sitting there with like a plugin ecosystem who were like, this is awful. Like what, like, why would you, why would anyone?…”

“MCP before we even open source it was born of the idea of like, I'm in a company that is growing crazy. I'm in the development side of things, development tooling side of things. I will grow slower than the rest. How can I build something that they can all bui…”

“Goose was the first open source agent interface or agent that reached out to us and worked with us to integrate MCP. And I think Rad is actually like technically the first non-anthropic contributor to MCP ever on like day two or something like that, like very,…”

“The first one is that on the MCP side, we always knew that we wanted to find a neutral home for MCP to make sure that the that the industry understands that this is stays open, that this is something safe to adopt.”

“You have Google, Microsoft, Amazon Block, Bloomberg, Cloudflare, OpenAI, Anthropic. Just a platinum member, create a foundation.”

“We got approached by our friends at block to discuss because they were looking into like donating goose, I think at the time. And so there was a question around doing something together. And then we approached open AI and they were very, very welcoming and lik…”

“Like for us, it's the first time we at Anthropic have an open source foundation.”

“Well, it does seem like basically a lot of the labs have kind of like converged onto some similar-ish way of doing RL, and they're all kind of back at the same level of like Frontier again”

“Like, one of the things that we saw in the hallucination rate is that Anthropoc's Claude models at the very left-hand side here with the lowest hallucination rates out of the models that we've evaluated Amnesians on.”

“Like for the first time you can actually get one shot data analysis, right? Which, you know, if you're going to do a customer database, analyze a cohort retention, right? That's just stuff that you had to do by hand before. And our team, the other, it was like…”

“It literally becomes an issue of like raise capital, turn that directly into growth, use that to raise three times more. And if you can keep doing that, you literally can outspend any company that's built. Not any company. You can outspend the aggregate of com…”

“I have a controversial opinion that Claude does not do RL on the agent swarms or agent team.”

“Anthropic doesn't just add two billion dollars of revenue in one month. You know, with, without having, you know, huge demand and they're doing it at positive margins, right?”

“OpenAI probably is hiring FDEs to go into the enterprise and then Anthropic is embedded at Goldman Sachs.”

“You can run Turbo Puffer either in SAS, right? That's what cursor does. You can run it in a single tenant cluster. So it's just you. That's what Notion does. And then you can run it in, in, in BYOC where everything is inside the customer's VPC. That's what, fo…”

“Big bang moment where things will accelerate so quickly that it becomes a self-reinforcing loop. And at that point it's sort of like off to the races and there will be no more like slowly catching up. You know, just have Claude being so good at everything.”

“The way I'm using it is I have co-work running, and I'm telling co-work, here's where I normally go every morning to find the latest bugs. Go read the entire bug list, separate out which ones are fixable, which ones are not fixable, and then for the fixable on…”

“We internally at Anthropica are now probably much closer to the point where, like, don't even write a memo. Just, like, build, like, let's build all the candidates very quickly. Let's just build all of them and then pick the best ones.”

“I've given it like pretty clear instructions about, okay, here are some people, if they book over other meetings, I'm probably going to go to their meeting. Like if Dario schedules a meeting. Not try to reschedule Dario.”

“Honestly though, like I see that you're using Opus 4.6, right? Like my recommendation for people is increasingly don't worry about it anymore. Just like tell it what you want it to do. And it's probably going to figure out a way to do it.”

“At Anthropic as a group of people, we're deeply worried about the impact that the tools are going to have on the labor market, especially for like junior employees that, because I think it's only honest to say that when we talk about automating a lot away, a l…”

“We're going to keep shipping things that we're going to keep shipping things that we're going to keep iterating on this thing like pretty quickly, but which I mean, you can sort of continue to expect that every single week, there's going to be like a small new…”

“I caught myself, like, starting to use Cowork for coding tasks, which is not ostensibly what we built it for, right?”

“Every single person who's like most amazed is usually amazed about a thing that I didn't even expect Cowork would be good at.”

“And in terms of architecture and UX and everything else that we've been working on Anthropic, it often is quite useful for you to like anthropomorphize cloud aggressively and just be like, this is a person. What would you do if you give, if you had a person, r…”

“The thing that ultimately led to skills is that we wanted to connect this little prototype to our data warehouse. And the team very quickly discovered that like, instead of building a custom tool for the thing to talk to our data warehouse, They just, like, ma…”

“And what cowork actually became is like, we sort of picked the right pieces out of the many prototypes that we had. Right. And that's maybe also like, I think an important qualifier whenever people mention this like 10 day number, I do think it's important to …”

“So on, on macOS, we use the Apple virtualization framework, which is pretty solidly optimized. Like it's good stuff.”

“Right when we launched Cowork, I think one of the users that went most viral on Twitter X was clean up your desktop, which is of course silly. That's such a smart thing, right? Like you don't need a model to clean up your desktop.”

“Cowork is cloud code running in a virtual machine with a little bit of padding, a little bit more guardrails, making it a little safer, a little bit more convenient for people who don't want to first open up the terminal when they go to work.”

“We built so much tech around Claude leaving useful comments inside a Google Doc, and now it just does it, just like leaves a comment in your Google Doc and that's how you interact with it.”

“I don't think we're far away from claw being very effective at like using your computer and not just a theoretical computer.”

“So like clock code is like quite optimized for coding tasks, and we mostly evaluate whether or not we're getting better or worse, depending on how good it is at like a typical sweet job. And cloud co-work on the other hand, we evaluate more against typical kno…”

“Last week we started writing a new networking service and networking driver. That optimizes how Claw talks to the internet.”

“We do tell co-work to make heavy use of the planning tool or to make heavy use of the ask user question tool, right? We do want it to come up with like, Different scenarios of, okay, tease out what the user actually wants. Don't go off to work for like four ho…”

“So we currently run like a, we currently run like a lightweight VM and we put clock code into the VM and we do that for a number of reasons. Safety and security is a big one, but even if you ignore for a second safety and security and you're just like, okay, Y…”

“One thing that is very fun for me about skills in particular is that they're so easy to make. Like anyone can make a skill, like a text message could be a skill and they can be so hyper-personalized to you.”

“The sort of the idea of a Labs team is that it should only work on things that make really no sense for anyone else to work on.”

“And then what we have, what we call Frontier Headroom evals, where we actively want to be at 30% pass rate. And that's actually been a effort that we took in partnership with Anthropic and OpenAI in the past maybe two or three months, because we actually hit a…”

“Sonic couldn't handle two tools with the same name in OpenAI, GPT, 5.2. It was like, ah, I can figure this out. So that was an interesting one that we learned by accident through a SEV.”

“I think if they if they had similar level of compute, they would be very competitive and maybe even beat the largest models, at least from what I've seen.”

“Like all the open source models, I think what happened was they got like very hyped and people were very interested in using them. But I think like over time, like there was a spike in usage for these models. And then it goes back to open AI, Anthropic and Goo…”

“Maybe 50% is still on Okunai. Yeah. Yeah. And then 50 on everything else. Yeah. But everything else is like mainly Anthropic and Google.”

“And then we did this for Opus. And it returned, like, yeah, it lied 10 times. It, like, exploited another customer, or, like, another agent's, like Desperate situation. It made price cartels like a hundred different, a hundred times. It like did all of this li…”

“So every single model from Anthropic since have been going in this direction. And I think one interesting thing is that like, OpenAI models don't. They, Quite plainly, they don't, they behave really well. And you know, you don't know if this is like, good, lik…”

“So we pitched it to the people we were already working with at Anthropic and they were like, yeah, you can have space. This sounds fun.”

“Anthropic was one of our early customers in doing evals, so we did, like, dangerous capability evals nothing we published openly”

“Claude is actually really, really lenient for tool calls. So even if, you know, your coding agent harness messes up, it can figure out that, oh, I'm being sent this error and can fix itself. Not the case with you know open models”

“We put a few hundred million dollars into Anthropic from our fund earlier this year.”

“Like, one of the reasons Anthropic has had extraordinary sort of velocity is because they picked the transform architecture and said, this is simple, let's double down on it, right? And now, luckily, there's enough investment going into space that we can affor…”

“And there, P zero from day one was coding. The reason the mechanism system there was, if we crack coding, Then we will crack AGI. You know, our mission is AGI. We want to get there safely. If we focus on coding, it's such a generally powerful capability that i…”

“Have you met Dario? Dario's a scientist. He's gone from zero to like what will soon be a trillion dollar company in four years.”

“What happened is, Anthropic basically achieved takeoff in October of last year. That training run.”

“Agent Labs get discounts from every model provider, and that's also very interesting when people compare public pricing of, like, a discounted cloud code from Anthopic versus what Anthopic does with Model Labs, with Agent Labs”

“I still do think they were bandwidth limited or they were compute limited because if you look at their quota and how aggressive and reducing it and using it it's gotten much better, but especially two months ago, I mean, you would burn through your quota in a …”

“The labs do not have 200 people dedicated to like, you know, being on call with you with Goldman Sachs going like, okay guys, what do you need? We got it. You need the Microsoft Teams zero integration. Got it. You don't use GitHub. You use this like weird org …”

“And the second was that reinforcement learning was going to be the biggest driver for LLM capabilities. Today, very obvious three years ago was not an opinion held or direction held at either OpenAI or Google or Anthropic or others.”