Aug 9, 2023 · 50m · mad
Startup to Industry Standard: Lukas Biewald Explains How W&B Scaled MLOps for OpenAI, NVIDIA & More
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, FirstMark Partner Matt Turck interviews Lukas Biewald, Co-Founder and CEO of Weights & Biases, about building a developer-first MLOps platform. Biewald discusses W&B's evolution, strategies for converting bottom-up developer adoption into enterprise contracts, and adapting ML infrastructure for Large Language Models.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 18.2% of the talking time here. How this is scored →
Matt as informed peer 3.0 Guest teaching 2.9 Guest disagreement 0.5 Matt pushing back 0.5
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 17:49 Debunking VC expectations on enterprise LLM adoption
Lukas directly pushes back on venture capital hype, explaining that despite the industry excitement, very few enterprises have successfully put LLMs into actual production.
Hardest push from Matt ▶ 37:44 Probing top-down enterprise execution behind bottom-up developer GTMMatt presses Lukas on the difficulty of scaling beyond bottom-up adoption, pointing out that developer virality alone rarely achieves full enterprise standardization without a structured sales force.
Biggest teaching moment ▶ 20:05 Correcting the concept of discrete model countsLukas educates Matt on modern ML architecture, explaining that asking how many models a company has in production misses the point since modern applications use interconnected webs of upstream models.
Matt holds his own ▶ 30:35 Highlighting specific developer marketing nuancesMatt displays deep familiarity with W&B's specific growth tactics by bringing up their open-author blog platform and how it engages community developers.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Lukas Biewald's Journey from CrowdFlower to Weights & Biases | 2 | 1 | 0 | 0 | Matt introduces Lukas and references their prior 2015 meeting alongside basic W&B funding numbers. Lukas provides a friendly historical summary of CrowdFlower and how developer frustration led to founding Weights & Biases. | |
| Defining the Evolving Machine Learning Developer Persona | 3 | 3 | 0 | 0 | Matt asks clarifying questions about what constitutes an ML developer and what W&B actually tracks. Lukas educates the host on the complexity of hyperparameters, pre-processing, artifacts, and model registries. | |
| Adapting MLOps Workflows for Large Language Models | 2 | 4 | 1 | 0 | Matt prompts Lukas on whether LLMs fundamentally change MLOps platforms. Lukas explains that while workflows remain exploratory, LLMs shift developer attention from complex loss graphs to qualitative text analysis and anecdotes. | |
| Challenges in Model Monitoring and Enterprise LLM Adoption | 3 | 4 | 2 | 1 | Matt asks if enterprises are deploying LLM applications into production at scale. Lukas offers a reality check, noting VCs are often surprised by how few enterprises have LLMs in true live production due to monitoring challenges. | |
| Real-World Enterprise ML Use Cases and Market Maturity | 3 | 4 | 1 | 1 | Matt asks for production model count estimates from sophisticated enterprise customers. Lukas reframes the premise by explaining that enterprise applications consist of deeply nested multi-model pipelines where counting individual models is misleading. | |
| Building a Developer-Centric Bottom-Up Go-To-Market Strategy | 4 | 2 | 0 | 1 | Matt demonstrates knowledge of developer GTM tactics by referencing W&B's open blog publishing platform and courseware. Lukas outlines their slow exponential growth, focus on NPS, and third-party integrations. | |
| Enterprise Sales Execution: Bridging Developer Usage to Organization Standardization | 4 | 2 | 0 | 1 | Matt asks how developer usage transitions into enterprise standardization and asks about optimal AE hiring profiles. Lukas details how credit card usage inside large companies leads to enterprise security and compliance deals. | |
| Executive Leadership Practices, Staying Technical, and Industry Tooling Analysis | 3 | 3 | 0 | 0 | Matt asks Lukas how he stays technical as CEO and requests reviews of popular developer tools. Lukas breaks down why PyTorch beat TensorFlow due to developer empathy rather than just feature sets. |
Statements from this episode (13)
Biewald: Data labeling software requires a top-down sales model
“Data labeling, I think really wants to be a top down sale”
Biewald: Almost all major LLMs were trained using Weights & Biases
“I think all of the major LLMs out there, almost all were trained using weights and biases.”
Biewald: LLM API developers are less mathematically specialized than traditional ML engineers
“Even the people Working with a lot of these APIs you know, are, like, less huge math nerds than, you know, some of the people that have been training, you know, models for a long time.”
Biewald: Developers building with LLMs focus heavily on qualitative anecdotes over metrics
“In the LL world, like the anecdote is something people really pay attention to.”
Biewald: Simple operational errors cause more model failures than data drift
“People talk a lot about data drift in the industry. And that's this idea that like, you know, like language changes over time and you want to know that it's changing and sort of like have your model you know, notice that and update it. But I guess like what I …”
Biewald: Most enterprises have not deployed LLMs into production yet
“I think that LLMs in particular, we talk to a lot of the people and we don't see a ton of people getting them into production yet. And I think it's funny, like VCs are always surprised, like when we tell them that I think that I don't know. I'm bullish on LMS,…”
Biewald: OpenAI is a W&B customer with a small number of production models
“OpenAI has been, like, a longtime customer. I mean, I consider them, like, extraordinarily sophisticated, and they have a pretty small number of models in, in production, so.”
Biewald: No single industry accounts for over 15% of W&B revenue
“No one vertical is more than like, you know, 15% of our usage or revenue or anything like that.”
Biewald: Traditional ML methods like boosted trees remain a large share of W&B usage
“A lot of people still running, you know, boosted trees or, you know, random forests inside of weights and biases. So we, you know, it's like actually huge. We should probably do a block, but it's still a big fraction of our You know, of our user base.”
Biewald estimates 99% of Global 2000 use ML for core operations
“I bet 99% of the global 2000 is using machine learning for something that they actually really care about.”
Biewald: Pharma is investing far more in deep learning than realized
“I think pharma is investing way more in deep learning than people realize.”
Biewald: Technical buyers don't want sales dinners, they just want facts
“Nobody wants to golf or anything. I mean, that's for sure, right? Like, I mean, there's people like a super aggressive salesperson. Some of my salespeople are really competitive. I am actually really competitive myself, but they kind of like suppress it in a w…”