Dec 18, 2025 · 48m · catalyst
Will inference move to the edge?
0:00 / 0:00
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of Catalyst, host Shail Khan and computer science professor Dr. Ben Lee examine whether AI inference workloads will shift from centralized hyperscale data centers to distributed edge infrastructure and explore the profound technical, latency, and power grid implications of this transition.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Shayle holds 37.3% of the talking time here. How this is scored →
Shayle as informed peer 4.5 Guest teaching 4.7 Guest disagreement 0.1 Shayle pushing back 0.4
speaking balance: gold is Shayle, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 32:37 Guest offers contrarian view on pipeline GPU saturation
The guest pushes back on assumptions of guaranteed edge construction by highlighting potential diminishing returns in model scaling and the likelihood of repurposing existing hyperscale capacity.
Hardest push from Shayle ▶ 42:10 Host demands clarification on the 80/20 local breakdownThe host refuses to accept the broad categorization of 'local compute' and actively pushes the guest to delineate between regional edge data centers and on-device hardware.
Biggest teaching moment ▶ 36:22 Guest explains parameter reduction and thermal barriersThe guest walks through the severe technical physics trade-offs of shrinking a 1-trillion parameter model down to a 7-billion parameter mobile model, detailing context memory and thermal constraints.
Shayle holds their own ▶ 44:50 Host connects edge distribution to aggregate energy increasesThe host demonstrates sharp analytical command of energy systems by pointing out that edge data center adoption will sacrifice hyperscale PUE efficiency and raise aggregate electricity consumption.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Shayle as informed peer | Guest teaching | Guest disagreement | Shayle pushing back | Why |
|---|---|---|---|---|---|---|
| Listener Survey Announcement and Gift Card Promotion | 0 | 0 | 0 | 0 | Introductory promotional announcement and sponsor ads read by a voiceover. No dialogue occurs between the host and guest. | |
| Framing the AI Compute and Grid Demand Dilemma | 0 | 0 | 0 | 0 | Solo host monologue framing the central thesis of the episode regarding AI compute growth, power grid bottlenecks, and edge inference potential. | |
| The Three Compute Tiers and Cloud Efficiency | 5 | 6 | 0 | 0 | The host asks the guest to define compute categories and recalls historical AV edge computing discussions, while the guest clearly explains hyperscale PUE metrics and hardware-sharing efficiencies. | |
| Model Training vs. Inference Energy Workloads | 4 | 6 | 0 | 0 | The host asks whether training compute will ever decentralize, and the guest details Meta research showing the three-way energy split between preprocessing, training, and inference. | |
| Edge Rationale, Latency Tolerances, and Cyber-Physical AI | 5 | 5 | 0 | 0 | The host notes how users tolerate latency in reasoning models like Deep Research and asks about robotics, which the guest categorizes under cyber-physical AI requiring strict responsiveness guarantees. | |
| Hardware Interconnects, Power Spikes, and DIDT Challenges | 6 | 6 | 0 | 1 | The host displays solid technical insight by pointing out the bizarre practice of dummy workloads to mitigate power spikes, which the guest validates and terms the DIDT challenge. | |
| Sponsor Break: Bloom Energy, Engie, and Energy Hub | 6 | 4 | 0 | 1 | Following the sponsor read, the host poses a detailed siting thought experiment comparing one 1-GW data center site against one hundred 10-MW sites, which the guest enthusiastically endorses. | |
| Geographic Clustering, Reliability, and Network Redundancy | 5 | 5 | 0 | 0 | The host queries why historical clustering occurred in regions like Northern Virginia, and the guest elaborates on internet exchange points, tax incentives, and workload rollover redundancy. | |
| CDN Precedents and Market Drivers for Edge AI | 5 | 6 | 1 | 1 | The host presses on why small edge inference sites are not yet being actively built, prompting the guest to draw parallels to Content Delivery Networks and Points of Presence. | |
| On-Device Inference: Privacy, Parameters, and Thermal Limits | 5 | 6 | 0 | 0 | The host brings up Apple as an obvious driver for on-device inference, and the guest breaks down the severe memory, parameter reduction, and battery/thermal constraints on consumer devices. | |
| The 2035 Prediction: The 80/20 Compute Distribution Rule | 6 | 6 | 0 | 2 | The host presses the guest for a concrete 2035 projection; the guest introduces the 80/20 rule, which the host immediately drills into to pin down the exact edge vs. device split. | |
| Systemic Energy Impacts and Autonomous Agentic Workloads | 7 | 6 | 0 | 0 | The host insightfully deduces that decentralized edge inference will actually increase total grid power consumption due to degraded PUE, which the guest confirms before exploring autonomous agent workloads. |
Statements from this episode (16)
Effectively 100% of global AI compute currently occurs in centralized cloud
“It's an energy question because the answer today is effectively a hundred percent in that first category, cloud.”
Google achieves a data center power usage effectiveness near 1.1
“Google's PUE is close to 1.1, which is to say for every watt going to compute, there's an additional .1 watts going to the overheads of power delivery or cooling or whatever.”
Meta study: AI energy costs split evenly across pre-processing, training, and inference
“There was a study we did when I was a visiting research scientist at Meta where we Found that energy costs for AI were roughly broken into three categories. There's a data pre-processing aspect as well, and that's about a third. The training is another third, …”
AI inference spending must surge to justify current adoption optimism
“If the optimism about AI is to be justified, you're going to have to see inference costs go way up because that will be an indicator that adoption has gone up in a fairly significant way, both among individual users, but also among companies and enterprise use…”
Generative AI is reconditioning users to tolerate multi-second latency delays
“What is interesting with generative AI is that we are being reconditioned to tolerate much longer delays. So if you use something like GPT or you use something like Claude or your favorite chatbot, oftentimes it's just sitting there thinking for seconds and se…”
Cyber-physical AI applications will require edge computing for low latency
“So I agree that there will be cases where we will need those really low latencies, and that is going to require edge computing much closer to the user, so we have much shorter internet delays, network delays.”
AI training compute cycles cause massive power fluctuations unlike inference workloads
“And some of the people in the energy space may know that there are massive energy fluctuations or power fluctuations we will see in data center usage when the GPUs go from this computational intensive phase where you're learning the model weights to this commu…”
AI training data centers run dummy workloads to flatten dangerous power spikes
“What they do in large part, because those spikes are actually problematic, not just to the grid, but to the equipment inside the data center as well. So what they do at least sometimes to manage that is they create dummy workloads. So they keep the power profi…”
LLM inference prompts are processed locally on one to eight GPUs
“When you send a prompt to for processing by a large language model, that prompt is probably handled by one GPU or maybe Eight GPUs inside a single machine. So, and the reason that is, is because the model sits in that machine, the data sits in that machine, an…”
Siting distributed edge compute will become easier than 1GW data centers
“I think finding capacity there may eventually become easier than finding the next thousand megawatts.”
If AI scaling slows, repurposed central GPUs will cannibalize edge data centers
“And if it turns out that maybe there are diminishing returns from training larger and larger models, or maybe we run out of data because we've exhausted all the data that's available on the internet. When those things happen, it may be that demand for these GP…”
Competition on AI latency will force local GPU deployments in major markets
“I think the catch there will be if one of these model providers or one of these application developers makes performance a distinguishing feature of their offering, right? If they start competing on performance rather than on capability, then we're going to se…”
Apple is the most likely company to move AI inference on-device
“It's not hard to picture that, like, if somebody's gonna move a lot of this inference on device, it's gonna be Apple.”
80% of AI inference compute will move to the edge by 2035
“So I would say that we could be getting 80% of our compute done locally and leaving 20% of the heavy lifting or the more esoteric, the more corner case compute for the data center cloud. That is, of course, excluding the training. The training will Continue to…”
Shifting inference to edge data centers will reduce efficiency and increase energy costs
“I think as you shrink the system down, you will get, you will lose an efficiency. You will be trying to build these 20 megawatt data centers and maybe footprints or facilities that weren't designed initially for those workloads. So yes, I think total energy co…”