Nov 23, 2015 · 25m · mad
Machine Learning in Production with Josh Bloom, Co-founder Wise.io
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
At DataDrivenNYC, Wise.io co-founder and UC Berkeley professor Josh Bloom explores the technical, strategic, and practical realities of deploying machine learning in production environments. He covers build-versus-buy decision criteria, technical debt, human-in-the-loop workflow augmentation, and strategies for designing fault-tolerant AI systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 5.3% of the talking time here. How this is scored →
Matt as informed peer 0.5 Guest teaching 2.8 Guest disagreement 0.7 Matt pushing back 0.2
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 5:50 Pissing off data scientists on accuracy trade-offs
Josh playfully acknowledges he might annoy audience members by arguing accuracy is often overprioritized over model interpretability.
Hardest push from Matt ▶ 17:21 Matt Turck probing buyer explainability requirementsMatt questions whether non-black-box explainability is actually a primary friction point when selling ML solutions into enterprises.
Biggest teaching moment ▶ 18:00 Josh Bloom reframing enterprise buyer ROI dynamicsJosh clarifies that enterprise buyers focus primarily on ROI upfront, and algorithm explainability only becomes a focus after ROI metrics are established.
Matt holds his own ▶ 17:21 Matt Turck framing enterprise sales realitiesMatt demonstrates industry familiarity by linking technical model interpretability directly to the operational mechanics of enterprise sales.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Defining the AI Build vs. Buy Decision | 0 | 2 | 0 | 0 | Josh Bloom delivers an uninterrupted monologue introducing Wise.io and framing the AI build versus buy decision. The host does not speak during this segment. | |
| Wise.io Architecture and Core ML Workflow | 0 | 2 | 0 | 0 | Bloom continues his presentation explaining Wise.io's engineering architecture and managed service strategy. The host remains silent throughout the segment. | |
| Multi-Axis Optimization & Model Interpretability | 0 | 3 | 1 | 0 | Bloom outlines trade-offs between model accuracy, interpretability, and execution speed, humorously noting he might annoy practitioners. The host does not participate. | |
| Machine Learning Technical Debt and Glue Code | 0 | 3 | 1 | 0 | Bloom discusses Google's technical debt paper and the prevalence of glue code in production ML systems. The host does not intervene. | |
| Fault-Tolerant ML and Augmentation Loops | 0 | 3 | 1 | 0 | Bloom details fault-tolerant machine learning and human-in-the-loop feedback mechanisms. The host remains silent. | |
| Real-World ML Failures and Concluding Remarks | 3 | 4 | 1 | 1 | Matt Turck transitions to Q&A, asking polite follow-up questions about model explainability in enterprise sales and user workflows. Bloom expands constructively on enterprise adoption dynamics. |
Statements from this episode (12)
Wise.io chose customer success as its first target for ML automation
“In an industry, the place where we wound up landing is in customer success.”
Wise.io chose managed cloud services over building infrastructure in-house
“So we made, as a young startup, the decision that if there was a managed service around what we needed to do to get something into production, and we're at the millions of predictions level a month over dozens of customers now, we were just going to, we were j…”
Wise.io serves millions of monthly predictions across dozens of customers
“And we're at the millions of predictions level a month over dozens of customers now”
Bloom: Very few data science teams prioritize model explainability
“Explainability or interpretability turned out to be a very, very important optimization that very few data science teams will be cognizant of unless they're really thinking about it.”
Bloom: Netflix couldn't deploy its $1M prize algorithm due to complexity
“They paid a million dollar bounty, and they wound up looking at the code, and there were hundreds of separate models that were then boosted together, and they said there's no way we can do it.”
Bloom: 95% of production machine learning code is just glue code
“Algorithms are important, but 95% of all machine learning code in production is actually just glue code. It's connecting all of these different pieces together.”
Bloom: ML platform decisions must center on production and maintenance costs
“The other ones that I think are critical when making those decisions about which platforms to use and implement within your own organizations is, what is that cost and time to actually put this into production? And then once you put it into production, what is…”
Bloom: Feedback loops transition ML from human augmentation to automation
“Over time, if you build the appropriate feedback loops into your systems, the system itself will wind up learning from those processes and get better and better, so you can actually start automating those processes.”
Wise.io uses agent acceptance and rejection of suggestions as model feedback
“What we do in Wise is we give, essentially, the agents the ability to take our suggestions for how to answer a support ticket, and if they don't, then that becomes feedback for us, and if they do, that also becomes feedback.”
Bloom: Netflix and Google build fault tolerance to handle ML errors
“Netflix and Google, some of the best machine learning companies in the world, and these are their core products, and they still make mistakes. Yet these are not fatal mistakes. They've built fault tolerance into the machine learning.”
Bloom: Support headcount still scales linearly with incoming ticket volume
“Support is still that last sort of place where you wind up having to scale the total number of people in your support system and support operations by the number of incoming tickets.”
Wise.io's ML automatically resolves 5% to 20% of support tickets
“Over time, the system becomes so confident in some fraction of the answers, Five to 10 to 20%. It can basically just answer them without any humans on our client side actually looking at it, and those tickets get solved and people are satisfied.”