Jun 25, 2024 · 1h 32m · latent-space
State of the Art: Training 70B LLMs on 10,000 H100 clusters
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space Podcast, Jonathan Frankle (Databricks) and Josh Albrecht (Imbue) discuss the low-level systems engineering, bare-metal hardware realities, and empirical optimization required to train 70B+ parameter foundation models. They share practical insights on cluster architecture, debugging distributed training bottlenecks, fixing flawed evaluation benchmarks, and building reliable code-centric AI agents.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 4.6 Guest teaching 5.7 Guest disagreement 1.4 The hosts pushing back 1.6
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:15:48 Jonathan's dismissal of Needle in a Haystack
Jonathan reacts with exasperation ('Oh, for the love of God'), arguing the benchmark fails to test holistic context utilization and measures nothing realistic.
Hardest push from the hosts ▶ 1:15:16 Swyx defending abstract evaluation testsSwyx pushes back against the guests' dismissal of abstract tests like ARC-AGI by pointing out humanity's successful reliance on SATs and IQ tests as capability predictors.
Biggest teaching moment ▶ 55:45 Josh explaining emergence as a metric mirageJosh systematically disproves the premise of sudden model emergence by explaining that discontinuous accuracy jumps are simply log-linear perplexity improvements passing threshold bands.
The host holds their own ▶ 37:49 Swyx challenging bare-metal abstraction choicesDrawing on his AWS background, Swyx challenges Imbue's custom orchestration layer by asserting that Kubernetes provides essential vendor-agnostic portability for scaling.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Introductions and Catching Up on the LLM Training Landscape | 5 | 3 | 1 | 2 | Swyx introduces the guests, tracks Databricks' recent developments, and asks clarifying questions regarding Shutterstock licensing arrangements and data provenance. | |
| DBRX Architecture, Dinosaur Mascots, and the Hair Dye Bet | 6 | 2 | 1 | 1 | Swyx recaps DBRX technical specs (132B total, 36B active, 12T tokens) while Jonathan jokes about team bets and dinosaur mascots. | |
| Imbue Open Sources the 70B Foundation Model Training Playbook | 4 | 6 | 1 | 1 | Josh outlines Imbue's 70B playbook release across infra, evals, and CARBS hyperparameter optimization; Jonathan strongly validates the release as rare and essential. | |
| Hardware Realities and Bizarre GPU Cluster Failure Modes | 4 | 6 | 1 | 1 | Jonathan and Josh share war stories about cluster failures, from stolen InfiniBand cables to memory correction latency and GPUs outputting incorrect arithmetic. | |
| Co-Designing a 4,096 H100 GPU Cluster with Voltage Park | 5 | 5 | 1 | 2 | Swyx inquires about the 4,096 H100 cluster topology and partnership with Voltage Park, prompting Josh to explain their non-standard 3-tier networking choices. | |
| Automated Bare-Metal Provisioning and Low-Level Hardware Health Checks | 3 | 7 | 1 | 1 | Josh walks through low-level automated bringup using Metal-as-a-Service, out-of-band management interfaces, and debugging directly with firmware vendors like Dell. | |
| Full-Stack Infrastructure: DIY Bare-Metal vs. Managed Cloud Platforms | 5 | 5 | 2 | 3 | Swyx compares Imbue's bare-metal approach with Mosaic's cloud abstraction, leading Jonathan to explain why even cloud users end up dealing with low-level physical realities. | |
| Cluster Reliability Metrics, Triage Automation, and Node Churn | 5 | 6 | 2 | 2 | Swyx presses on cluster failure rates; Josh corrects the uniform failure assumption and details automated dmesg anomaly detection on node reboots. | |
| Cluster Bringup Logistics: Small Teams, Cable Management, and Physical Assembly | 4 | 6 | 1 | 1 | Josh details the physical bringup of 24,000 cable connections executed by a small core team in coordination with vendor technicians. | |
| Distributed Training Tools, S3 Local Mirrors, and Uber's Kraken Registry | 5 | 6 | 1 | 1 | Josh discusses memory fragmentation jitter and highlights simple tooling choices like BitTorrent-based Kraken image distribution over heavy cloud filesystems. | |
| Contrasting Infrastructure Philosophies: Minimal Abstractions vs. Standardized APIs | 6 | 5 | 2 | 4 | Swyx pushes back from an AWS perspective arguing Kubernetes standardizes infrastructure, but Josh counters that Kubernetes is mismatched with training job failures. | |
| Training Dynamics: MoE Network Bandwidth, HSDP, and Next-Gen Interconnects | 5 | 7 | 1 | 1 | Jonathan explains how MoE training stresses network bandwidth, requiring HSDP, and compares Google's 3D torus TPU design with Nvidia's grouped NVLink topologies. | |
| Imbue's Specialized Focus on Code, Reasoning, and Text-Only Architectures | 4 | 5 | 1 | 1 | Swyx probes Imbue's decision to focus purely on text and coding rather than vision; Josh explains how agent reasoning can outsource multimodal components. | |
| Evaluation Metric Precision: Multiple-Choice Perplexity and Data Mix Schedules | 4 | 6 | 3 | 1 | Josh explains how CARBS optimizes hyperparameters across cost scales; Jonathan and Josh cordially disagree on whether dynamic data mix schedules provide real gains. | |
| The Mirage of Emergence: Log-Scale Perplexity vs. Metric Discontinuities | 4 | 8 | 2 | 2 | Swyx asks whether emergence breaks CARBS scaling laws; Josh schools on how emergence is an artifact of non-linear accuracy metrics rather than log-scale loss. | |
| Cleaning Open-Source Benchmarks and Addressing Question Ambiguity | 4 | 7 | 1 | 1 | Josh breaks down human annotation efforts to clean benchmark ambiguity, demonstrating that top models reach near total saturation on well-formed questions. | |
| Programmatic Benchmarks, Code Understanding, and Realistic Test Coverage | 5 | 6 | 1 | 1 | Swyx questions the scope of code understanding; Josh and Jonathan discuss SWE-bench and the difficulty of evaluating nuanced code without leaking bad patches. | |
| Navigating Imperfect Evaluations and Modeling Uncertainty in AI Agents | 4 | 7 | 1 | 1 | Jonathan reflects on living with imperfect evaluations when directing research, while Josh outlines Imbue's work on detecting question ambiguity and agent uncertainty. | |
| Evaluating Abstract Reasoning (ARC-AGI) vs. Pragmatic Business Problems | 6 | 6 | 2 | 4 | Josh and Jonathan critique abstract reasoning benchmarks like ARC-AGI as synthetic toys, prompting Swyx to defend standardized abstract testing as historically predictive. | |
| The Limitations of 'Needle in a Haystack' and Long-Context Agent Scenarios | 4 | 8 | 4 | 1 | Jonathan vehemently dismisses Needle in a Haystack as disconnected from real-world utility, while Josh explains why long context requires active relevance filtering. | |
| Code Execution as the Universal Agent Interface vs. Hard-Coded Tool Calling | 5 | 6 | 1 | 1 | Josh argues that general code execution is vastly superior to rigid tool-calling schemas, while Jonathan highlights the value of native SQL execution for structured enterprise data. | |
| Knowledge Graphs, Data Schemas, and Bounding Agent Action Spaces | 5 | 5 | 1 | 2 | Swyx prompts on whether knowledge graphs offer an enduring abstraction; Josh and Jonathan agree they fit specific ontologies but boundary control remains challenging. | |
| Future Roadmaps: Databricks Science Sharing and the Abra-Kadabra Model Lineage | 4 | 4 | 1 | 1 | Josh outlines Imbue's roadmap toward practical coding agents while Jonathan hints at future DBRX evolutionary models (Abra, Kadabra, and Alakazam) before closing. |
Statements from this episode (43)
Frankle: Databricks released text-to-image model with Shutterstock
“Is that we finally released our text image model which has been a year in the making through a collaboration directly with Shutterstock.”
Frankle: OpenAI, Google, Meta, and Apple have data deals with Shutterstock
“And you know, I, at least I've heard in the news, like opening, I Google, Meta Apple have all called Shutterstock and made those deals.”
Frankle: Databricks model is uniquely trained purely on Shutterstock data
“So a lot of models have had Shutterstock data incorporated into them, but this is the only model I know of so far where it was, you know, exclusively and specifically trained just on the vanilla Shutterstock data. There was nothing else mixed in. You know, we …”
Albrecht: Imbue will not release model weights, but will open-source training tools
“We're not releasing the model. We're not releasing the weights, but we are releasing a bunch of different things that should make it easier for other people to make their own models.”
Albrecht: Imbue is releasing a new reasoning benchmark and 11 cleaned evaluations
“We're releasing a whole bunch of different data there, a new benchmark about code, reasoning, understanding, as well as our own private versions of 11 different open source benchmarks. So things like PoolQ or ANLI, where we've gone through and kind of cleaned …”
Imbue is releasing approximately 450,000 human evaluation judgments
“A final thing that we're releasing there is around 450,000 human judgments about ambiguity and question quality, which we used In the process of cleaning these evaluations”
Frankle: Fault tolerance is missing from fundamental model training primitives
“Fault tolerance is still not really built into any of the fundamental primitives of training models. And so if something breaks, you have to go figure out what broke your job stops. You have to restart your job. It is a nightmare just to get to the point where…”
Frankle: Porch pirates stole MosaicML InfiniBand cables twice before data center delivery
“Our InfiniBand cables getting stolen from the data center twice, like in boxes before they arrived at the data center, like, you know, porch pirate basically had stolen our InfiniBand cables back when those were hard to come by”
Frankle: Most AI data centers are retrofitted, not built for high heat
“In data centers that for the most part were not built remotely for this kind of power or heat and have been retrofitted for this. Like failures happen on a good day with normal CPUs. And this is not a good day and not a normal CPU for the most part.”
Albrecht: 4K GPU clusters require 3-tier networking versus standard 1K 2-tier setups
“The normal, the like vanilla setup or, you know, these large clusters as vanilla as it can be is what's normally like a 127 node cluster. So closer to like 10, 24 GPUs instead of 4000. Here we have a larger cluster. As you start to get into the larger clusters…”
Albrecht: Training on AWS prevents diagnosing low-level hardware errors
“And if we're just using, you know, AWS or some other cloud provider, These errors are still going to be there, and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong.”
Frankle: Large-scale model training forces teams to debug the full infrastructure stack
“It's kind of impossible if you're doing training to not go all the way through the entire stack, regardless of what happens. Like somehow I'm still chatting with cloud providers about power contracts, even though the whole point of dealing with the cloud provi…”
Albrecht: Imbue's GPU cluster failure rate is well below industry 3% benchmark
“The number that we've heard from other people is like they're having about three percent. I don't think we're experiencing failure rates that are that high. I think ours is actually quite a bit lower than that, probably because we've taken the time to like dig…”
Albrecht: Imbue Manages Infrastructure with Three to Six Engineers
“Like our infrastructure team is like
You know, it fluctuates from week to week, depending on like how many things are on fire and how much we need to build.
But it's like between like three and six people, like it's small.
It's not like some huge team of like …”
Albrecht: 4,000-GPU Three-Tier Cluster Requires 12,000 Cables and 24,000 Plugs
“Like to bring up this cluster you know, with 4000 GPUs and three tier networking, networking architecture, you have 12,000 cables.
So that's 24,000 things that need to be plugged in.”
Albrecht: Unsynchronized garbage collection across distributed nodes steadily degrades cluster MFU
“Because you have hundreds of machines, they're doing garbage collection at slightly different times, and then they get slightly further apart, and slightly more and more jittered, until eventually they're all happening kind of at random times, and just, like, …”
Albrecht: Imbue avoids Kubernetes to keep cluster infrastructure simple to debug
“Less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes, for example, I would just directly launch these things and it's just been much easier to debug this way.”
Frankle: Databricks runs across six or seven different cloud providers
“Think we're running on like six or seven different clouds right now.”
Frankle: Training MoE models with FSDP creates severe network bandwidth bottlenecks
“And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP zero three style. Where there's just a lot of parameters getting shuffled back and forth and your ratio of kind of compute to amount of dat…”
Frankle: Networking was the hardest part of training DBRX at scale
“And so actually the networking part of DPRX was the single hardest thing. I think of the entire process, just get MOE training, working at scale across a big cluster.”
Frankle: Google TPUs provide much higher bandwidth-to-compute ratios
“TPUs have a very different network bandwidth to compute ratio. They have a lot more bandwidth just objectively and TPUs per chip tend to be a little bit less compute intensive and have a little bit less memory.”
Albrecht: Vision is not essential for most coding and reasoning agent tasks
“And actually we found that for most of the kind of like code writing and reasoning problems that we care about, the visual part isn't really a huge important part of it.”
Albrecht: CARBS models compute cost per sample for hyperparameter search
“CARBS is, it's maybe a backronym, but it's for Cost Aware Pareto Region Bayesian Search... The point is that it's a cost aware hyperparameter tuner. So most hyperparameter tuners you kind of say, okay, here's this objective function. I want you to make this nu…”
Albrecht: Cost-aware tuning reveals scaling laws for all hyperparameters
“So by doing that, we can see the scaling laws or not just, you know, the scaling laws from like the, you know, chinchilla paper, the scaling laws for all parameters. We can see how does the number of layers change with this? How does the You know, the learning…”
Albrecht: Dynamic data mix schedules during LLM training yield negligible gains
“We did some experiments and we've actually talked to a bunch of researchers who were doing work here as well and looking at kind of their experiments on this. And we were originally pretty hopeful because it sounds like something that should work and make sens…”
Frankle: Dynamic data mixing during pre-training is effective for domain-specific models
“We've had some surprisingly good luck with this. We just released a paper on it. The details matter a lot and it really matters what you're trying to do with the model. But it's been quite effective for us depending on the setting. And certainly when we're thi…”
Albrecht: LLM emergence is an artifact of non-linear evaluation metrics
“This emergent behavior that you're seeing, Is not really emergent behavior, but is really a function of the evaluation metrics that we're using.”
Frankle: Deep learning log scales can make trends look however you want
“Anything can look however you want it to look if you put it on a log scale to a certain extent. And log, we love our log scales and deep learning for various reasons. Everything looks very clean on a log scale until everything looks very flat on a log scale.”
Albrecht: Imbue reproduced 500-1,000 examples per dataset to stop eval contamination
“Let's just reproduce, you know, 500 to a thousand examples for every single one of these data sets ourselves and just make sure that this data is definitely not in the, you know, the training set. So we did that and then we're able to like now be confident abo…”
Albrecht: Benchmark performance differences vanish once ambiguous questions are cleaned
“The main takeaway from any of the, like, actual performance is like, once you fix these ambiguous examples, a lot of these benchmarks are really saturated. Like, I think it's important to look at like, you know, like when you're talking about performance on NL…”
Albrecht: AgentBench paper's appendix examples are actually incorrect solutions
“Like we were looking at the agent bench paper, I think just last week for our paper club. And one of the things that we noticed is that actually like both of the examples in the appendix that are given as like traces where it got it right. This is actually not…”
Frankle: Top AI scientists must tolerate broken infrastructure and imperfect evals
“Like the most successful scientists I see are the ones who are okay operating in a world where everything's going to be broken. And yet we can still cobble things together and make something interesting happen.”
Albrecht: Coding agents communicating uncertainty are far more useful than slightly more accurate ones
“I would much rather have a coding agent that will give me back a thing. And you know, it's actually the code doesn't work like 10% less of the time than some other model, but it will tell me a hundred percent of the time. When it got like when it's not sure, l…”
Albrecht: Optimizing for competitive coding benchmarks does not create useful programmers
“Like, we do a lot of code generation, but we don't really do a lot on, like, code competition problems for the very, very hard ones, so that you can go very far down that route and make something like really good at those problems, but not actually that useful…”
Frankle: No Databricks enterprise customer asks for abstract reasoning AI
“I don't think I have a single customer that's asking to, you know, have AI solve abstract reasoning problems.”
Frankle: Multimodal AI models will inevitably require massive context windows
“Once you get into multimodal land, you're just going to end up with giant context. It's kind of unavoidable.”
Frankle: Needle in a Haystack eval fails to measure holistic context usage
“I think the problems with needle in a haystack are well known. You know, it doesn't measure anything real. You're not even testing the model's ability to holistically use the context just to identify one part of the context. So you can do some wacky things to …”
Albrecht: Code execution expands agent capabilities far beyond hard-coded tool calling
“Instead of worrying about like weird hard coded agents using tools, Like let's just make them able to actually write code robustly and make that code work and be able to debug that code, know if that code is safe to run, like get really good at the like code w…”
Frankle: PDF parsing remains an unsolved problem in 2024
“PDF parsing is still an unsolved problem, even in 20, 24.”
Frankle: Text-to-SQL is one of the most impactful LLM use cases for enterprise
“Like it's, you know, text to SQL is still, or like having a model be able to make SQL calls in the backend is actually like one of the single most useful things for my customers. It sounds really boring. Models are really good at it and it moves the needle day…”
Albrecht: Messy real-world data limits knowledge graphs to niche problems
“But I think in the real world, it gets a lot messier than like knowledge graph style of things where it's like, well, is there a relationship between these two nodes? Like, ah, I don't know. Like is, are these two separate nodes? Like those kinds of messy bord…”
Frankle: Releasing Open-Source Models Is Not Databricks' Core Bread and Butter
“Releasing models open source is not our day-to-day bread and butter. It's kind of a fun reward that we get to do sometimes when we have something really cool to share and a little bit of time and spare GPUs in our hands. But for the most part, everything is go…”