Sep 30, 2025 · 1h 4m · y-combinator
Anthropic Head of Pretraining on Scaling Laws, Compute, and the Future of AI · Y Combinator
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Y Combinator podcast interview, host Ankit Gupta speaks with Nick Joseph, Head of Pre-Training at Anthropic, about the fundamental principles, hardware infrastructure, organizational strategies, and alignment challenges of building frontier AI models. Nick shares detailed insights on scaling laws, cluster engineering, web data dynamics, and the key technical skills needed to build superintelligent systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The partners as informed peer 3.6 Guest teaching 4.3 Guest disagreement 1.4 The partners pushing back 0.9
speaking balance: gold is the partners, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 13:10 Nick dismissing academic skepticism of scaling laws
Nick candidly rejects the culture of academic research labs like FAIR, asserting that external critics missed the big picture and made nonsensical arguments against obvious empirical scaling laws.
Hardest push from the partners ▶ 32:42 Ankit challenging data scarcity with PageRank filteringAnkit pushes back against the premise that finding useful internet data is unsolvable by proposing that standard PageRank thresholding should cleanly separate useful training data from web noise.
Biggest teaching moment ▶ 32:55 Nick explaining why web graph metrics fail for AI trainingNick educates the host on why PageRank and click-based graph metrics fail to capture training utility, explaining that the true frontier lies in the unlinked long tail of internet text.
The partners hold their own ▶ 55:49 Ankit challenging compute scaling with architectural optimizationsAnkit demonstrates deep industry awareness by citing specific architectural modifications from open-source Chinese labs, such as attention caching and custom attention kernels, to challenge whether scaling alone suffices.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The partners as informed peer | Guest teaching | Guest disagreement | The partners pushing back | Why |
|---|---|---|---|---|---|---|
| Nick Joseph's Background: Vicarious, OpenAI, and Early AI Safety | 3 | 3 | 1 | 0 | Ankit Gupta opens the interview exploring Nick Joseph's background across GiveWell, Vicarious, and OpenAI. Nick explains his transition from economics and charity evaluation to AI safety. | |
| Defining Pre-Training, Next-Word Prediction, and Scaling Laws | 4 | 4 | 1 | 1 | Ankit asks why autoregressive next-token prediction won out over masked language modeling like BERT. Nick emphasizes that autoregressive training allows direct sampling for products and provides an empirical scaling objective. | |
| Scaling Laws, Hyperparameters, and Small-Scale Testing | 4 | 4 | 1 | 1 | Ankit explores hyperparameter selection and neural architecture search. Nick explains how empirical scaling laws reveal power-law curves and how scaling compute dominates hyperparameter fine-tuning. | |
| Early Infrastructure, Cloud Providers, and Cluster Topologies | 4 | 5 | 2 | 1 | Ankit asks about Anthropic's early infrastructure when it was a small team. Nick reveals that they wrote custom all-reduce operations because they aimed to scale beyond what existing frameworks like PyTorch supported at Facebook. | |
| Scaling Conviction vs. Academic Research Lab Culture | 4 | 5 | 3 | 1 | Nick expresses surprise at external skepticism toward scaling laws, criticizing academic research culture at places like FAIR for ignoring empirical scaling. He then details hardware profiling and Model FLOPs Utilization (MFU). | |
| Pair Programming, Internal Knowledge, and Debugging Tools | 4 | 4 | 1 | 1 | Nick describes learning profiling and debugging through pair programming with Tom Brown and Sam McCandlish. He contrasts specialist and generalist staffing strategies in pre-training teams. | |
| Distributed Failure Domains and Data Center Scale Challenges | 4 | 5 | 1 | 1 | Ankit asks about unexpected challenges at scale. Nick highlights single failure domains in distributed training runs and how hardware components in massive clusters frequently break. | |
| TPU vs. GPU Trade-offs in Pre-Training and Inference | 4 | 5 | 1 | 1 | Nick breaks down hardware bottlenecks, explaining why inference is high-bandwidth memory bound while pre-training is compute-bound. He discusses building minimal reproducers to collaborate with cloud providers on low-level bugs. | |
| The Evolution of Post-Training, RL, and Organizational Dynamics | 4 | 4 | 1 | 1 | Ankit inquires whether the rise of reinforcement learning shifts focus away from pre-training. Nick explains that RL introduces its own scaling laws and stresses the importance of avoiding organizational competition between teams. | |
| Internet Data Limits, PageRank, and Finding Quality Data | 4 | 6 | 3 | 2 | Ankit suggests PageRank could filter high-quality internet data. Nick pushes back, explaining that web popularity metrics do not correlate with what an AI model needs to learn long-tail knowledge. | |
| Synthetic Data, Distillation Risks, and Web Data Contamination | 4 | 4 | 1 | 1 | Ankit and Nick discuss synthetic data distillation and the risks of recursive training on LLM-generated web content, illustrated by how false data distributions compound. | |
| Evaluating Frontier Models, Loss Metrics, and Domain Benchmarks | 4 | 4 | 1 | 1 | Ankit discusses difficult evaluation domains like medical AI conversations. Nick argues that driving down next-token loss on high-quality doctor-patient transcripts remains an underappreciated and effective benchmark. | |
| Defining AI Alignment, Steering Models, and Democratic Values | 3 | 4 | 1 | 1 | Nick defines AI alignment both in terms of long-term AGI goal-sharing and near-term persona steering via Constitutional AI, using the analogy of building a steering wheel before choosing where to drive. | |
| Integrating Alignment into Pre-Training vs. Fast Post-Training Loops | 3 | 4 | 1 | 1 | Nick explains why alignment is predominantly handled in post-training due to fast iteration cycles, while noting that fundamental robustness may eventually require embedding alignment signals directly into pre-training. | |
| Future AI Paradigm Shifts and the Risks of Unnoticed Machine Learning Bugs | 3 | 4 | 1 | 1 | Nick discusses future risks, noting that unnoticed low-level machine learning bugs are what scare him most because they can derail entire multi-month training runs. | |
| Full-Stack Debugging: From High-Level ML Dynamics to Network Packets | 4 | 5 | 1 | 1 | Ankit describes classic architecture wiring bugs. Nick expands on the challenge of full-stack debugging, from high-level learning dynamics down to low-level kernel precision casting and network packet protocols. | |
| Hiring Strategy: Systems Engineers, Physicists, and Deep Technical Problem Solvers | 3 | 4 | 2 | 1 | Nick dispels the idea that pre-training teams are primarily ML research PhDs, emphasizing that frontier labs critically need systems engineers and problem-solving physicists capable of deep technical debugging. | |
| Alternative Architectures vs. Reliable Scaling of Transformers | 4 | 5 | 2 | 1 | Ankit brings up alternative architectures and optimizations from open-source labs. Nick explains that reliable transformer scaling consistently outperforms novel architectures, and highlights how pre-training co-designs models around inference constraints. | |
| Scaling Compute and the Challenges of Infinite Resources | 4 | 4 | 1 | 1 | Ankit and Nick discuss operating under 100x compute scaling. Nick highlights startup opportunities in automated cluster chip verification and cautions against building heavy scaffolding around current model limitations. | |
| Career Advice for Entering the AI Workforce | 2 | 3 | 1 | 0 | Nick offers career advice for entering the AI industry, recommending that newcomers focus on deep systems engineering and preparing for post-AGI governance rather than classical ML math. |
Statements from this episode (38)
Nick Joseph: Early AI safety discourse was largely theoretical and philosophical
“At the time, a lot of the AI safety discussion was kind of theoretical, like the models weren't actually that good. They weren't really posing these dangers, so it was a lot more like philosophical.”
Nick Joseph followed OpenAI safety leads to join Anthropic at founding
“Basically everyone I worked with, like all the safety leads left, which yeah, invited me to go to Anthropic, and that was sort of the reason I joined OpenAI, was because I cared about AI safety and wanted to work with them. So then I went with them to join Ant…”
Joseph: AI scaling laws predictably quantify loss reductions from compute and data
“There's this idea of scaling laws, which is that you can actually quantify, like, as you put in more compute, more, more data, more parameters, you get models in a very, you got a lower loss, a better prediction of the next word in a very predictable way.”
Joseph: AI progress relies on a revenue-to-compute positive feedback loop
“There's this positive feedback loop where you can train a model, You can use it to make something useful and sell that and get more money, use that to buy more compute, and then use that to train a better model. And we've sort of run that cycle over and over a…”
Anthropic's Joseph: Compute matters far more than pre-training objective details
“I think that, like, the one sort of general intuition I have is, like, compute is the thing that matters. So, like, I think if you throw enough compute at any of these objectives, you're gonna get something that's probably pretty good, and can kind of be fine …”
Joseph: Public estimates placed GPT-3 training cost at $5 million
“Like the public estimates for GP three, I remember, were that it cost five million dollars to train, which you're like, on the one hand, five million is kind of a lot, but it's like a lot for an individual person. It's not really a lot from like a company pers…”
Anthropic reverse-engineered cloud provider GPU physical room layouts to fix latency
“We're using a cloud provider, but I think it's kind of, it's not actually that different, because one of the things that was surprising to me is you actually have to understand the literal layout. Like, I remember at 1.1 of my co-workers running a clustering a…”
Anthropic built custom distributed training to scale beyond Facebook's infrastructure
“We don't want to outsource this to some package because A, we're about to go to a bigger scale, like PyTorch, for instance, they had a package for doing this. But we were going to go to a bigger scale than Facebook had been to. And you don't want to have a dep…”
Joseph: Original scaling laws paper spanned 11 orders of magnitude
“Like, you know, the scaling, I think the original scaling laws paper had, like, 11 orders of magnitude, and there was, like, this intense debate on whether it would continue for, like, another point.”
Joseph: LLM training requires collaborative infrastructure work over publishable research papers
“And to do a project like training a large language model requires a lot of people to collaborate on like a really complicated piece of infrastructure that isn't going to be a paper, right? Like you're not going to publish like, oh, I got a slightly, I got five…”
Joseph: Anthropic had to hack PyTorch profiler for large-scale GPU clusters
“The PyTorch profiler was, like, pretty good, actually, throughout for a single GPU. You want to, like, profile a GPU, the PyTorch profile would work. But if you wanted to profile a job on 100,000 of GPUs, that, like, hadn't really been done much, and then that…”
Joseph: Too many specialists forces managers to connect cross-domain optimizations
“I think if you get too many people who are specialists, you end up with a lot of effort has to come from the manager, from like the lead to connect everything, and to notice something like, Ah, if we change the architecture here, that would make this, like, ef…”
Joseph: Standard AI chip parallelization makes the entire cluster one failure domain
“Like one thing that I think is like the standard way people parallelize chips isn't the whole thing is one failure domain. Like one chip fails, the whole thing can crash.”
Joseph: Inference needs more HBM bandwidth while pre-training is FLOPs-intensive
“Inference as a workload in general. Tends to require more HBM bandwidth. You end up doing you sort of the simplest form of sampling since you're going one at a time. You have to load all the weights for every token. And that means you might want a lot of HBM b…”
Joseph: Using multiple AI chip architectures multiplies engineering workload
“The downside of having multiple chips is that you have to write the thing multiple times. In theory, you could have abstractions across them, but they're different enough that it's pretty hard to do that. So you can sort of end up, if you do all the workloads …”
Joseph: Reinforcement learning exhibits scaling laws where compute yields better models
“You can get pretty big wins from RL. You sort of have another set of scaling laws. It's like you put more and more compute into RL, you can get better and better models out of that.”
Joseph: AI theory is usually wrong and progress must be empirical
“I think almost everything kind of has to be done empirically. Like, you can kind of, like, come up with theories, but in practice, like, the first thing you're gonna do with your theory is test it, and most of the time you'll have gotten it wrong. So you shoul…”
Joseph: Anthropic avoids pre-training and post-training rivalry seen at other labs
“Yeah, I think we managed to avoid this, and it's pretty collaborative. Like, we're basically all producing one model and kind of can, but I do think at other places there's been some, from what I've heard, there's been some amount of, like, friction between, b…”
Joseph: Sparsely linked long-tail data may be most valuable for frontier AI
“And it might be that like, that data ends up more valuable because you, everything that's linked to a lot, you've already got. Like at some point, you're maybe like going for the tails, or you're going for the stuff that no one's ever, like, you know, it's onl…”
Joseph: Training purely on raw LLM generations cannot produce a better model
“Theoretically, I shouldn't be able to train a better model than that. Like, I'm just going to get the same thing out. So I think that's-”
Joseph: Adversaries actively publish web data designed to poison AI models
“There are people who are, like, trying to put stuff out that is, like, as damaging as possible for the model, you know, how can I make it past the filter and get into the model would be totally like secretly useless.”
Nick Joseph: Solving coding interview questions proved to be shockingly narrow, not AGI
“I used to think that if you had an AI that could solve coding interview questions, it would probably be AGI. I was like, that's what I did to get my job, I could probably do the job. And it turns out like, nope, nope, you solve those, it's shockingly narrow, a…”
Nick Joseph: Third parties can steer frontier AI labs by publishing evals
“Like, it is the case that, like, the labs right now are really driven by getting good eval scores. And it's hard to make them, and anyone can do it. There's no comparative advantage to having the model to making an eval. So I do think it's actually, like, an i…”
Joseph: Sci-fi underestimates AGI because AI software can be copied a billion times
“Sci-fi movies actually, like, underestimate the impact of it. Like, you always have this, like, one robot that's, like, a human. And I'm like, well, wouldn't you have, like, a billion of them? Like, you can just copy them everywhere. So you should picture, lik…”
Joseph: Training safety into models is more robust than using system prompts
“I think you get different amounts of robustness if it's trained into the model versus if it's in a prompt that you can, like, add or remove or tell, like, ignore all previous instructions, that sort of thing.”
Joseph: Technical AI steerability must precede debates over whose values to embed
“I think that's a really hard problem. I think it's, like, actually kind of downstream of being able to pick any. I think of it almost, I think one analogy I've heard that I like is, like, putting a steering wheel on a car. It's like, if you don't have a steeri…”
Anthropic's Joseph: Do Everything Possible in Post-Training Over Pre-Training
“The way I usually think about it is anything you can do in post training, you probably should, because your iteration loop, like the ability to make progress is really fast. You can try something, you can try it again, you can try it again.”
Anthropic's Joseph: Certain AI Alignment Pieces Will Move to Pre-Training
“I do think at some point there will be, like, some pieces of alignment that, like, you do want to export back into pre-training because that might be a way to, like, Put them in with more strength, like, more robustness, kind of, or more core to the intelligen…”
Anthropic's Joseph: Scaling alone likely will not achieve AGI without further paradigm shifts
“Like I think the sort of shift towards more RL is like one paradigm shift in the field, and I think it's, I think there will probably be more. I think a lot of people sort of argue about like, oh, it's like, you know, current paradigm's enough to get us to EGI…”
Anthropic's Joseph: A single undetected bug can derail model training for months
“A single bug can like, Derail you for months. Yeah. And when you think about it, like you, the models take months to train. So you can kind of like lose a whole generation off of something that just looks like, ah, you know, it turns out like this piece of you…”
Anthropic's Joseph: Very few engineers can debug ML from math to bytes
“I think one thing that's, like, surprisingly hard and there's very few people who can do is, like, kind of own that whole stack from, like, I understand how the ML is supposed to work and what the learning dynamics are, all the way down to, like, I know the by…”
Anthropic's Nick Joseph: Frontier pre-training teams primarily need engineers, not researchers
“The thing we, like, most need is engineers. Almost always, like, throughout, like, the entire history of this field. It's, like, the case that you throw more compute, the thing kind of works. The challenge is, like, actually doing that.”
Nick Joseph: Anthropic hired theoretical physicists who learned programming on the job
“Early on, we hired a lot of people from just, like, all sorts of backgrounds, and I think that people who are just smart and work really hard can learn this pretty fast, but you have to, like, want to. We heard a lot of physicists, for instance. Like theoretic…”
Nick Joseph: Scaling standard models is easier and more reliable than inventing novel architectures
“It's just that scale is easier, and it's more reliable, and I think you, we're still seeing really big gains to that.”
Anthropic's Nick Joseph: Pre-training decisions determine whether inference can run fast
“Oh, no, I think a ton about inference, because it basically, like, The problem inference is solving, like, we basically determine the problem inference is solving. We give them a model, and they have to, like, run that fast, and it's very easy to give them a m…”
Anthropic's rate limits are caused by short-notice compute shortages
“Anthropic has rate limits constantly, and people complain about it a lot, and like the reason is like, there's only so much compute we can get on short notice, so you, like, making your inference more efficient is like the way you can serve more users.”
Anthropic: Frontier AI models are first-shot attempts due to chip limits
“But I do think the change is massive, and I think people, like, don't realize how chip-limited AI, like, research is, or something right now, like, the models that everyone uses, right? If you're using, like, Cloud Sonic four, Cloud Opus four, it's like, it's …”
Anthropic's Joseph: Startups shouldn't over-invest in scaffolding future models won't need
“I think maybe the thing to watch out for is things where, like, they work now with a huge amount of work, like, to build up a scaffold, but the next generation, you're not gonna need the whole scaffold you built up.”