Nov 25, 2024 · 55m · latent-space
Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode, Fireworks AI CEO Lin Qiao sits down with Alessio Fanelli and Swyx to discuss the technical architecture of distributed inference, the shift toward Compound AI and declarative reasoning, and why open-source AI infrastructure is poised to outperform closed-source alternatives.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 22.7% of the talking time here. How this is scored →
The hosts as informed peer 4.9 Guest teaching 5.7 Guest disagreement 1.5 The hosts pushing back 2.2
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 46:54 Lin rejects competitor benchmarking tactics
Lin strongly criticizes a competitor publicly calling out Fireworks by name with biased benchmarks, insisting evaluations must be independently validated.
Hardest push from the hosts ▶ 34:09 Swyx invokes the Bitter Lesson against compound AISwyx refuses the premise that domain-specialized models have lasting moats, arguing a 10x larger model trained on generalized data will invalidate narrow expert models.
Biggest teaching moment ▶ 27:20 Lin contrasts declarative and imperative AI designLin systematically educates the hosts using relational database SQL optimization analogies to explain why declarative LLM orchestration will win developer adoption over brittle DAG pipelines.
The host holds their own ▶ 50:45 Swyx itemizes OpenAI's multi-billion dollar cost breakdownSwyx demonstrates deep domain financial knowledge by rattling off the exact line items of OpenAI's training, inference compute, and research amortization expenses.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Lin Qiao's Background at Meta and PyTorch Evolution | 4 | 7 | 1 | 1 | Swyx sets up the prehistory of PyTorch and Meta's early GenAI efforts, referencing past guest Soumith Chintala. Lin delivers an extensive masterclass on scaling PyTorch from research to production across Meta's ubiquitous systems. | |
| Pivoting from Horizontal PyTorch Cloud to Generative AI Inference | 4 | 6 | 1 | 1 | Swyx probes Fireworks' initial pivot from a horizontal PyTorch cloud to inference hosting. Lin explains the market dynamics of 2022, detailing why inference scales with the global population whereas training only scales with researchers. | |
| Focusing on App Developers and the Emergence of Llama Stack | 5 | 5 | 2 | 4 | Swyx pushes back on Meta's Llama Stack, expressing skepticism that developer adoption will follow simply because Llama is open source. Lin acknowledges it is very early and emphasizes Fireworks' role in delivering direct community feedback. | |
| The Concept of Compound AI and Multi-Modal Optimization | 4 | 7 | 1 | 1 | Swyx asks why Fireworks leaned heavily into Compound AI post-Series B. Lin articulates the architectural shift from one-size-fits-all distributed inference to multi-modal compound systems and three-dimensional optimization across quality, latency, and cost. | |
| Distributed Inference Architecture and Custom Kernel Acceleration | 5 | 6 | 2 | 3 | Alessio and Swyx press Lin on what distributed inference actually entails versus raw GPU rental moats. Lin details custom CUDA kernels, disaggregated execution, regional routing, and hardware specialization across heterogeneous architectures. | |
| Declarative AI Architecture and Fireworks' New Reasoning Model | 5 | 7 | 1 | 1 | Lin uses database SQL analogies to explain declarative AI system architectures over complex DAG pipelines. She reveals Fireworks' upcoming reasoning model trained to approach o1 reasoning quality. | |
| Inference Scaling Laws, Model Specialization, and the Bitter Lesson | 7 | 6 | 3 | 6 | Swyx directly challenges compound specialist models using Rich Sutton's Bitter Lesson, arguing larger general models eventually subsume narrow experts. Lin counters with human societal specialization and the shift toward test-time inference scaling laws. | |
| Fireworks Engineering Culture and the Cursor Partnership Case Study | 6 | 5 | 1 | 2 | Swyx and Lin discuss Cursor's fast-apply speculative decoding pipeline. Lin details how co-engineering high-throughput inference stacks with aggressive developer teams validated their Fire Optimizer product. | |
| Quantization Nuances, Open-Source Economics, and Multi-LoRA Serving | 7 | 6 | 3 | 3 | Swyx brings up public benchmark callouts and breaks down OpenAI's P&L compute amortization. Lin criticizes unfair competitor benchmarking and explains Fireworks' multi-LoRA architecture sharing base model memory across hundreds of adapters. | |
| Community Feedback, Discord Office Hours, and Hiring Expansion | 2 | 2 | 0 | 0 | Standard wrap-up segment with Lin inviting developer feedback on Discord and announcing engineering hiring across multiple disciplines. |
Statements from this episode (13)
PyTorch was originally built for researchers without considering production requirements
“PyTorch actually started as the framework for researchers. Don't care about production at all.”
AI inference matters more than training because it scales with global population
“Our prediction is for those kind of applications, the inference is much more important than training. Because inference scale is proportional to the upliminal world population. And training. Training scale is proportional to the number of researchers.”
Shawn Wang doubts Meta's Llama Stack will win broad developer adoption
“I've been a little bit more doubtful on Lama stack. I think you've been more positive. Basically, it's just like the meta version of whatever HuggingFace offers, you know, or TensorRT, or BLM, or whatever the open source opportunity is. But like, to me, it's n…”
Customer inference workloads rarely align with foundation model training distributions
“The data distribution in their inference workload doesn't align with the data distribution in the training data for the model, right? It's a given, actually. If you think about this, because researchers have to guesstimate what is important, what's not importa…”
Compelling GenAI applications require compound AI systems spanning multiple modalities
“In order to really build a compelling application on top of JNI, we need a compound AI system. Compact AI system basically is going to have multiple models across modalities along with APIs, whether it's public APIs, internal proprietary APIs, storage systems,…”
Fireworks AI runs custom acceleration kernels for almost all served models
“For almost for all models, for all large language models, all your models.”
Fireworks AI will release a reasoning model inspired by OpenAI's o1
“So another announcement is we will also announce a, our next
Declarative system is going to be appear as a model that has extremely high quality, and this model is inspired by O-one announcement from OpenAI.
You should see that by the time we announce this o…”
Specialized open-source expert models will outperform one-size-fits-all closed-source models
“And that's our prediction is With specialization, there will be a lot of expert models, really, really good, and even better than, like, one size fits all open source closed source model.”
Pre-training on human data is hitting limits; synthetic data is required
“So I think on the data side, we're approaching the limit and the only data to increase that is synthetic generated data.”
Fireworks AI deployed a custom workload optimization stack specifically for Cursor
“We have a unique automation stack that is one size fits one. We actually deploy to cursor early on. Basically optimize for their specific workload, and that's a lot of juice to extract out of there, and we see success in, in that product is actually can be wid…”
Fireworks AI serves fine-tuned LoRA adapters at base model pricing
“We wrote multi LoRa last year, actually, and we actually have this function for a long time and many people have been using it, but it's not well known that, oh, if you find your model, you don't need to use on demand. If you find your model is LoRa. You can u…”
Fireworks AI's multi-LoRA system serves up to 1,000 adapters per base model
“One base model can sustain a hundred to a thousand LoRa adapters. And then basically all these different LoRa adapters can share the same, like direct the same traffic to the same base model where base model is dominating the cost.”