The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 38 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 1 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Anthropic's Joseph: Compute matters far more than pre-training objective details
“I think that, like, the one sort of general intuition I have is, like, compute is the thing that matters. So, like, I think if you throw enough compute at any of these objectives, you're gonna get something that's probably pretty good, and can kind of be fine …”
Joseph: LLM training requires collaborative infrastructure work over publishable research papers
“And to do a project like training a large language model requires a lot of people to collaborate on like a really complicated piece of infrastructure that isn't going to be a paper, right? Like you're not going to publish like, oh, I got a slightly, I got five…”
Joseph: Sparsely linked long-tail data may be most valuable for frontier AI
“And it might be that like, that data ends up more valuable because you, everything that's linked to a lot, you've already got. Like at some point, you're maybe like going for the tails, or you're going for the stuff that no one's ever, like, you know, it's onl…”
Joseph: Training purely on raw LLM generations cannot produce a better model
“Theoretically, I shouldn't be able to train a better model than that. Like, I'm just going to get the same thing out. So I think that's-”
Nick Joseph: Third parties can steer frontier AI labs by publishing evals
“Like, it is the case that, like, the labs right now are really driven by getting good eval scores. And it's hard to make them, and anyone can do it. There's no comparative advantage to having the model to making an eval. So I do think it's actually, like, an i…”
Anthropic's Joseph: Certain AI Alignment Pieces Will Move to Pre-Training
“I do think at some point there will be, like, some pieces of alignment that, like, you do want to export back into pre-training because that might be a way to, like, Put them in with more strength, like, more robustness, kind of, or more core to the intelligen…”
Anthropic's Joseph: Scaling alone likely will not achieve AGI without further paradigm shifts
“Like I think the sort of shift towards more RL is like one paradigm shift in the field, and I think it's, I think there will probably be more. I think a lot of people sort of argue about like, oh, it's like, you know, current paradigm's enough to get us to EGI…”
Anthropic's Nick Joseph: Frontier pre-training teams primarily need engineers, not researchers
“The thing we, like, most need is engineers. Almost always, like, throughout, like, the entire history of this field. It's, like, the case that you throw more compute, the thing kind of works. The challenge is, like, actually doing that.”
Nick Joseph: Scaling standard models is easier and more reliable than inventing novel architectures
“It's just that scale is easier, and it's more reliable, and I think you, we're still seeing really big gains to that.”
Anthropic built custom distributed training to scale beyond Facebook's infrastructure
“We don't want to outsource this to some package because A, we're about to go to a bigger scale, like PyTorch, for instance, they had a package for doing this. But we were going to go to a bigger scale than Facebook had been to. And you don't want to have a dep…”
Joseph: Reinforcement learning exhibits scaling laws where compute yields better models
“You can get pretty big wins from RL. You sort of have another set of scaling laws. It's like you put more and more compute into RL, you can get better and better models out of that.”
Joseph: Anthropic avoids pre-training and post-training rivalry seen at other labs
“Yeah, I think we managed to avoid this, and it's pretty collaborative. Like, we're basically all producing one model and kind of can, but I do think at other places there's been some, from what I've heard, there's been some amount of, like, friction between, b…”
Joseph: Adversaries actively publish web data designed to poison AI models
“There are people who are, like, trying to put stuff out that is, like, as damaging as possible for the model, you know, how can I make it past the filter and get into the model would be totally like secretly useless.”
Nick Joseph: Solving coding interview questions proved to be shockingly narrow, not AGI
“I used to think that if you had an AI that could solve coding interview questions, it would probably be AGI. I was like, that's what I did to get my job, I could probably do the job. And it turns out like, nope, nope, you solve those, it's shockingly narrow, a…”
Joseph: Technical AI steerability must precede debates over whose values to embed
“I think that's a really hard problem. I think it's, like, actually kind of downstream of being able to pick any. I think of it almost, I think one analogy I've heard that I like is, like, putting a steering wheel on a car. It's like, if you don't have a steeri…”
Anthropic's rate limits are caused by short-notice compute shortages
“Anthropic has rate limits constantly, and people complain about it a lot, and like the reason is like, there's only so much compute we can get on short notice, so you, like, making your inference more efficient is like the way you can serve more users.”
Anthropic: Frontier AI models are first-shot attempts due to chip limits
“But I do think the change is massive, and I think people, like, don't realize how chip-limited AI, like, research is, or something right now, like, the models that everyone uses, right? If you're using, like, Cloud Sonic four, Cloud Opus four, it's like, it's …”
Anthropic's Joseph: Startups shouldn't over-invest in scaffolding future models won't need
“I think maybe the thing to watch out for is things where, like, they work now with a huge amount of work, like, to build up a scaffold, but the next generation, you're not gonna need the whole scaffold you built up.”
Joseph: AI scaling laws predictably quantify loss reductions from compute and data
“There's this idea of scaling laws, which is that you can actually quantify, like, as you put in more compute, more, more data, more parameters, you get models in a very, you got a lower loss, a better prediction of the next word in a very predictable way.”
Joseph: AI progress relies on a revenue-to-compute positive feedback loop
“There's this positive feedback loop where you can train a model, You can use it to make something useful and sell that and get more money, use that to buy more compute, and then use that to train a better model. And we've sort of run that cycle over and over a…”
Anthropic reverse-engineered cloud provider GPU physical room layouts to fix latency
“We're using a cloud provider, but I think it's kind of, it's not actually that different, because one of the things that was surprising to me is you actually have to understand the literal layout. Like, I remember at 1.1 of my co-workers running a clustering a…”
Joseph: Original scaling laws paper spanned 11 orders of magnitude
“Like, you know, the scaling, I think the original scaling laws paper had, like, 11 orders of magnitude, and there was, like, this intense debate on whether it would continue for, like, another point.”
Joseph: Too many specialists forces managers to connect cross-domain optimizations
“I think if you get too many people who are specialists, you end up with a lot of effort has to come from the manager, from like the lead to connect everything, and to notice something like, Ah, if we change the architecture here, that would make this, like, ef…”
Joseph: Using multiple AI chip architectures multiplies engineering workload
“The downside of having multiple chips is that you have to write the thing multiple times. In theory, you could have abstractions across them, but they're different enough that it's pretty hard to do that. So you can sort of end up, if you do all the workloads …”
Joseph: AI theory is usually wrong and progress must be empirical
“I think almost everything kind of has to be done empirically. Like, you can kind of, like, come up with theories, but in practice, like, the first thing you're gonna do with your theory is test it, and most of the time you'll have gotten it wrong. So you shoul…”
Joseph: Sci-fi underestimates AGI because AI software can be copied a billion times
“Sci-fi movies actually, like, underestimate the impact of it. Like, you always have this, like, one robot that's, like, a human. And I'm like, well, wouldn't you have, like, a billion of them? Like, you can just copy them everywhere. So you should picture, lik…”
Joseph: Training safety into models is more robust than using system prompts
“I think you get different amounts of robustness if it's trained into the model versus if it's in a prompt that you can, like, add or remove or tell, like, ignore all previous instructions, that sort of thing.”
Anthropic's Joseph: Do Everything Possible in Post-Training Over Pre-Training
“The way I usually think about it is anything you can do in post training, you probably should, because your iteration loop, like the ability to make progress is really fast. You can try something, you can try it again, you can try it again.”
Anthropic's Joseph: Very few engineers can debug ML from math to bytes
“I think one thing that's, like, surprisingly hard and there's very few people who can do is, like, kind of own that whole stack from, like, I understand how the ML is supposed to work and what the learning dynamics are, all the way down to, like, I know the by…”
Nick Joseph: Anthropic hired theoretical physicists who learned programming on the job
“Early on, we hired a lot of people from just, like, all sorts of backgrounds, and I think that people who are just smart and work really hard can learn this pretty fast, but you have to, like, want to. We heard a lot of physicists, for instance. Like theoretic…”
Nick Joseph: Early AI safety discourse was largely theoretical and philosophical
“At the time, a lot of the AI safety discussion was kind of theoretical, like the models weren't actually that good. They weren't really posing these dangers, so it was a lot more like philosophical.”
Nick Joseph followed OpenAI safety leads to join Anthropic at founding
“Basically everyone I worked with, like all the safety leads left, which yeah, invited me to go to Anthropic, and that was sort of the reason I joined OpenAI, was because I cared about AI safety and wanted to work with them. So then I went with them to join Ant…”
Joseph: Public estimates placed GPT-3 training cost at $5 million
“Like the public estimates for GP three, I remember, were that it cost five million dollars to train, which you're like, on the one hand, five million is kind of a lot, but it's like a lot for an individual person. It's not really a lot from like a company pers…”
Joseph: Anthropic had to hack PyTorch profiler for large-scale GPU clusters
“The PyTorch profiler was, like, pretty good, actually, throughout for a single GPU. You want to, like, profile a GPU, the PyTorch profile would work. But if you wanted to profile a job on 100,000 of GPUs, that, like, hadn't really been done much, and then that…”
Joseph: Standard AI chip parallelization makes the entire cluster one failure domain
“Like one thing that I think is like the standard way people parallelize chips isn't the whole thing is one failure domain. Like one chip fails, the whole thing can crash.”
Joseph: Inference needs more HBM bandwidth while pre-training is FLOPs-intensive
“Inference as a workload in general. Tends to require more HBM bandwidth. You end up doing you sort of the simplest form of sampling since you're going one at a time. You have to load all the weights for every token. And that means you might want a lot of HBM b…”
Anthropic's Joseph: A single undetected bug can derail model training for months
“A single bug can like, Derail you for months. Yeah. And when you think about it, like you, the models take months to train. So you can kind of like lose a whole generation off of something that just looks like, ah, you know, it turns out like this piece of you…”
Anthropic's Nick Joseph: Pre-training decisions determine whether inference can run fast
“Oh, no, I think a ton about inference, because it basically, like, The problem inference is solving, like, we basically determine the problem inference is solving. We give them a model, and they have to, like, run that fast, and it's very easy to give them a m…”