Jan 1, 2025 · 1h 51m · latent-space
2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In their milestone 100th episode, Latent Space co-hosts Alessio Fanelli and Swyx deliver a comprehensive 2024 retrospective analyzing the rise of AI engineering, test-time reasoning compute, the Four Wars of AI, and dramatic inference cost deflation.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 99.9% of the talking time here. How this is scored →
The hosts as informed peer 6.1 Guest teaching 0.8 Guest disagreement 0.9 The hosts pushing back 0.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 48:53 Alessio defends Midjourney workflow over raw API models
Alessio sharply dismisses Swyx's 'skill issue' jab regarding Flux by framing usability and time efficiency as a Black Forest failure.
Hardest push from the hosts ▶ 15:10 Swyx rejects the open source convergence narrativeSwyx directly calls out saturation charts being near 100% as misleading evidence that open source models are catching up to o1.
Biggest teaching moment ▶ 1:01:15 Swyx breaks down memory vs vector database architectureSwyx systematically clarifies why memory requires interaction history, decay rates, and sleep consolidation rather than standard RAG vector lookups.
The host holds their own ▶ 11:20 Swyx details the frontier model pricing shift and market shareSwyx demonstrates deep domain mastery by detailing exact Ramp and OpenRouter dataset metrics showing OpenAI's market share dropping from 95% to 50-75%.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Celebrating 100 Episodes and the AI Engineer Era | 6 | 0 | 0 | 0 | Alessio and Swyx open their 100th episode by celebrating the validation of the AI Engineer movement and discussing its industry recognition via Gartner's hype curve. | |
| Research vs Production and the Big Scaling Debate at NeurIPS | 7 | 1 | 1 | 1 | Swyx recounts the NeurIPS debates on pre-training walls, contrasting test-time compute with inference-time compute optimality alongside papers from Chinchilla to Noam Brown. | |
| Frontier Model Competition, Market Shifts, and Inference Economics | 8 | 1 | 2 | 2 | Swyx challenges claims that open source is closing the gap with frontier reasoning models, dissecting market share shifts across OpenAI, Anthropic, and Gemini. | |
| Agent Frontiers, Systems ML, and Ilya's Keynote Insights | 7 | 1 | 1 | 1 | The hosts discuss agent frontier challenges, Jeff Dean's systems ML approaches with AlphaChip, and Ilya Sutskever's historical scaling predictions. | |
| Long-Tail NeurIPS Discoveries: Steganography and Dataset Trends | 7 | 0 | 0 | 0 | Swyx outlines long-tail research from NeurIPS, including DeepMind's steganography paper on agent collusion and the shift toward dataset tracks. | |
| The Four Wars of AI: Data Rights, Synthetic Data, and Reasoning Moats | 7 | 1 | 1 | 1 | The hosts analyze the data wars, synthetic reasoning distillation via STaR methods, and the durability of reasoning moats like o1. | |
| The Compute War: GPU Super-Rich Clusters vs GPU-Poor Practicality | 7 | 1 | 1 | 1 | Swyx introduces the GPU smiling curve, comparing 100k-cluster mega-labs with lean customer-facing wrappers like Suno and Bolt. | |
| The Multimodal War: Specialist Startups vs Integrated God Models | 7 | 2 | 2 | 2 | Swyx and Alessio debate specialist multi-modal startups against integrated omni-models, bantering over Flux, Midjourney, and Sora. | |
| The LLMOS & Agent War: Sandboxes, Memory Architecture, and Protocols | 8 | 2 | 2 | 2 | Alessio evaluates PyPI metrics for LangChain and CrewAI while Swyx elaborates on why memory architectures need temporal decay and sleep consolidation. | |
| The Shifting Evaluation Landscape: Benchmarks and Capabilities Tiering | 8 | 1 | 1 | 1 | Swyx reviews benchmark degradation from MMLU and GPQA to SWE-bench verified, setting up the framework for mature versus emerging capabilities. | |
| Editing Room Interlude: Capability Breakdown and the Price Collapse | 0 | 0 | 0 | 0 | Swyx delivers an editing room monologue charting the 3-orders-of-magnitude price-intelligence collapse throughout 2024. | |
| 2024 Chronological AI Rewind: January to June Milestones | 0 | 0 | 0 | 0 | Swyx continues his monologue rewind from January to June, touching on Perplexity, Devin's launch, Suno/Udio, Llama 3, and GPT-4o. | |
| 2024 Chronological AI Rewind: July to December & The Rise of Agents | 8 | 1 | 1 | 1 | The hosts reunite to discuss SSI's single-product mission, o1's launch velocity, Canvas vs Google Docs, and how AI will establish workplace skill floors. | |
| Top Latent Space Episodes of 2024 and Reflections | 6 | 0 | 0 | 0 | Alessio and Swyx wrap up with reflections on their top 2024 episodes, the lack of a real AI winter, and community growth. |
Statements from this episode (29)
Swyx: AI field will invert from research-heavy to engineering-heavy
“I think the AI world, the ML world is still very much research heavy. And that's as it should be because ML is very much in a research phase. But as we move this entire field into production, I think that ratio inverts into becoming more engineering heavy.”
Swyx: Inference compute optimality creates different scaling laws than Chinchilla
“Chinchilla paper is compute optimal training, but what is not stated in there is it's pre-trained compute optimal training. And once you start caring about inference, compute optimal training, you have a different scaling law and in a way that we did not know …”
Swyx: Gemini Flash accounts for 50% of OpenRouter requests
“Gemini Flash, according to Open Router, is now 50% of their Open Router requests.”
Swyx: OpenAI production market share dropped from 95% to 50–75%
“Basically over the course of 23, going into 24, OpenAI has gone from 95 market share to reasonably somewhere between 50 to 75 market share.”
Swyx predicts OpenAI will launch a $2,000 per month ChatGPT tier
“I think that 2000 dollars ChatGPT will come.”
Swyx: AI VCs and content creators should spend $1,000 monthly on AI tools
“I think if your job is a, at least AI content creator, or VC, or, you know, someone who, whose job it is to stay on top of things, you should already be spending like a thousand dollars a month on, on stuff.”
Swyx: OpenHands is ranked number one on SWE-bench Full
“He started open hand is currently still number one on SweetBench full, which is the hardest one.”
Fanelli: Agents extracting undocumented business processes will unlock enterprise adoption
“The agents are, that most people are building, Are good at following instruction, but are not as good as like extracting them from you. Yeah. So I think that will be a big unlock.”
Swyx: Model papers at NeurIPS are dead as research shifts to datasets
“The focus I saw is that model papers at NeurIPS are kind of dead. No one really presents models anymore. It's just data sets because it's all the grad students are working on.”
Swyx: Frontier labs view new learning rate optimizers as unimportant
“Most people at the big labs who I asked about this say that it's cute, but it's not something that matters.”
Swyx: Claude Sonnet and Gemini Outperform o1-Preview in Coding
“Claude Sonnet so far is beating O-one on coding tasks without At least one preview without being a reasoning model and same for Gemini pro or Gemini two point O.”
Swyx: VC appetite for GPU-rich early-stage startups is completely gone
“The appetite for GPU rich startups, like the, you know, the funding plan is we will raise sixty million and we'll give 50 of that to Nvidia. That is gone, right? Like no one's pitching that. This was literally the plan, the exact plan of like, I can name like …”
Swyx: AI models hit a 2T parameter wall, won't reach 10T
“As far as everyone is concerned, Claude, you know, Opus 3.5 is not coming out. GPT 4.5 is not coming out. And Gemini two, like we don't have pro whatever we've hit that wall, whatever that wall is. Maybe I'll call it like the two trillion parameter wall. Like …”
Swyx: Suno grew from zero to $20M ARR running on Modal
“Suno ramp has rated as one of the top ranked fastest growing startups of the year. I think the last public number is like zero to twenty million this year in ARR and Suno runs on Moto. So Suno itself is not GPU rich, but they're just doing the training on, on …”
Swyx: Claude wrapper Bolt.new reached $20M ARR
“The other one would be Bolt. There's a straight quad wrapper. And again, another now they've announced twenty million ARR, which is another step up from our eight million that we put on the title.”
Swyx: The 'GPU smiling curve' rewards infrastructure and user-facing apps over middle-tier models
“I kind of call this the GPU smiling curve where the edges do well, cause you're either close to the machines and you're like number one on the machines, or you're like close to the customers and you're number one on the customer side. And the people who are in…”
Swyx: Recraft V3 has overtaken Flux 1.1 as the top image model
“Recraft V three is now
being Flux 1.1, which is very surprising because Flux and Black Forest Labs are the old stable diffusion crew who left Stability after the management issues. So ReCraft has come from nowhere to be the top image model.”
Swyx: AI video generation needs five more years to become fully SOTA
“And so the researchers that I talked to are already starting to talk about that as the next frontier, but there's still maybe like five more years of video left. To actually be soda.”
Swyx: DeepMind has roughly a four-year advantage over OpenAI in world modeling
“So like they have maybe four years advantage on world modeling that OpenAI does not have. Cause OpenAI basically only started Diffusion Transformers last year when they hired Build Peebles. So DeepMind has a bit of advantage here.”
Fanelli: LangChain and LlamaIndex usage grows on PyPI while CrewAI stays flat
“So if you look at, you know, like chain still growing. These are the last, last six months. Lama index still growing. What I've basically seen is like things that one, obviously these things have a commercial product. So there's like people buying this and sti…”
Swyx: Diff mode will become the norm for AI code tools in 2025
“Canvas has incorporated the diff mode that both Anthropic and OpenAI and Fireworks has now shipped that I think is going to be the norm for next year, that everyone Need some kind of diff mode code interpreter thing.”
Swyx: SWE-bench resolution rates surged from 13% to ~50% in 2024
“Keep in mind, we started the year at 13%. And so now we're about 50 open hands is around there.”
Swyx: Frontier AI labs distinguish themselves by adopting new benchmarks
“The labs that are not that frontier will keep measuring themselves on last year's benchmarks. And then the labs that are actually frontier will tell you about benchmarks you've never heard of.”
Swyx: Iso-ELO LLM token costs dropped nearly 1,000x in 2024
“For the same amount of ELO, what you used to pay at the start of 24 you know, let's say, you know, 50, 40 to 50 dollars Per million tokens now is available approximately at, with Amazon Nova approximately at, I don't know, 0.075 dollars per token. So like seve…”
Swyx: Apple Intelligence is the largest transformer rollout since Google's BERT
“It is the probably the largest scale rollout of transformers yet after Google rolled out BERT for search”
Swyx: Always-on vision AI assistants will dominate desktop software by late 2025
“And like this time next year, I would be willing to bet that I would just have this running on my machine. And you know, I think That assistance always on that you can talk to with vision that sees what you're seeing. I think that is where at least one hour so…”