Nov 23, 2015 · 22m · mad

Black Boxes and Unicorns - DataRobot CEO Jeremy Achin

Jeremy Achin · 18m spoken Matt Turck · 59s spoken
0:00 / 0:00
▶ Watch on YouTube →

gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions

In this FirstMark DataDrivenNYC presentation, DataRobot CEO Jeremy Achin presents 'Black Boxes and Unicorns,' explaining how algorithm-agnostic interpretability tools demystify complex machine learning models while automated platforms and practical education can solve the data science talent shortage.

How this conversation actually went

Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 4.6% of the talking time here. How this is scored →

Matt as informed peer 0.5 Guest teaching 4.0 Guest disagreement 2.8 Matt pushing back 0.3
05100:0010:0020:000:48–4:26 · Matt as informed peer 0/10 Jeremy Achin Background and DataRobot Company Overview Achin provides his background as an insurance data scientist and Kaggle competitor while introducing DataRobot and citing Leo Breiman's 2001 paper on statistical modeling. Because this is a keynote monologue, the host does not participate.4:26–6:34 · Matt as informed peer 0/10 Redefining Black Boxes and Model Interpretability Achin aggressively redefines 'black box' as a term used by people afraid of technology who want to maintain 20-year-old habits. He outlines technical criteria for model transparency that apply across complex ensemble algorithms.6:34–10:25 · Matt as informed peer 0/10 Case Study: Hospital Readmission and Partial Dependence Achin presents a hospital readmission case study to demonstrate how partial dependence plots reveal marginal non-linear feature effects in complex models. He explains how this approach provides model interpretability without relying on simple linear regression.10:25–14:03 · Matt as informed peer 0/10 Critiquing Compliance and Arbitrary Representations of Reality Achin heavily critiques traditional statisticians and compliance teams, defining compliance as a last resort excuse to defend linear regression status quo. He uses humorous parables like statisticians in a bar and killer potatoes to show how regression coefficients can create arbitrary representations of reality.14:03–17:24 · Matt as informed peer 0/10 Democratizing Data Science and Presentation Takeaways Achin critiques traditional data science education pipelines for dropping students before teaching practical application. He argues that automated modeling tools will solve the data scientist shortage by enabling practical application first.17:24–22:57 · Matt as informed peer 3/10 Host Q&A and Event Conclusion Host Matt Turck opens Q&A by concisely summarizing DataRobot's automated model selection value proposition and inviting Achin to explain the product. Achin and the audience discuss Kaggle performance and unexpected algorithmic results.0:48–4:26 · Guest teaching 2/10 Jeremy Achin Background and DataRobot Company Overview Achin provides his background as an insurance data scientist and Kaggle competitor while introducing DataRobot and citing Leo Breiman's 2001 paper on statistical modeling. Because this is a keynote monologue, the host does not participate.4:26–6:34 · Guest teaching 4/10 Redefining Black Boxes and Model Interpretability Achin aggressively redefines 'black box' as a term used by people afraid of technology who want to maintain 20-year-old habits. He outlines technical criteria for model transparency that apply across complex ensemble algorithms.6:34–10:25 · Guest teaching 5/10 Case Study: Hospital Readmission and Partial Dependence Achin presents a hospital readmission case study to demonstrate how partial dependence plots reveal marginal non-linear feature effects in complex models. He explains how this approach provides model interpretability without relying on simple linear regression.10:25–14:03 · Guest teaching 6/10 Critiquing Compliance and Arbitrary Representations of Reality Achin heavily critiques traditional statisticians and compliance teams, defining compliance as a last resort excuse to defend linear regression status quo. He uses humorous parables like statisticians in a bar and killer potatoes to show how regression coefficients can create arbitrary representations of reality.14:03–17:24 · Guest teaching 4/10 Democratizing Data Science and Presentation Takeaways Achin critiques traditional data science education pipelines for dropping students before teaching practical application. He argues that automated modeling tools will solve the data scientist shortage by enabling practical application first.17:24–22:57 · Guest teaching 3/10 Host Q&A and Event Conclusion Host Matt Turck opens Q&A by concisely summarizing DataRobot's automated model selection value proposition and inviting Achin to explain the product. Achin and the audience discuss Kaggle performance and unexpected algorithmic results.0:48–4:26 · Guest disagreement 1/10 Jeremy Achin Background and DataRobot Company Overview Achin provides his background as an insurance data scientist and Kaggle competitor while introducing DataRobot and citing Leo Breiman's 2001 paper on statistical modeling. Because this is a keynote monologue, the host does not participate.4:26–6:34 · Guest disagreement 4/10 Redefining Black Boxes and Model Interpretability Achin aggressively redefines 'black box' as a term used by people afraid of technology who want to maintain 20-year-old habits. He outlines technical criteria for model transparency that apply across complex ensemble algorithms.6:34–10:25 · Guest disagreement 1/10 Case Study: Hospital Readmission and Partial Dependence Achin presents a hospital readmission case study to demonstrate how partial dependence plots reveal marginal non-linear feature effects in complex models. He explains how this approach provides model interpretability without relying on simple linear regression.10:25–14:03 · Guest disagreement 7/10 Critiquing Compliance and Arbitrary Representations of Reality Achin heavily critiques traditional statisticians and compliance teams, defining compliance as a last resort excuse to defend linear regression status quo. He uses humorous parables like statisticians in a bar and killer potatoes to show how regression coefficients can create arbitrary representations of reality.14:03–17:24 · Guest disagreement 3/10 Democratizing Data Science and Presentation Takeaways Achin critiques traditional data science education pipelines for dropping students before teaching practical application. He argues that automated modeling tools will solve the data scientist shortage by enabling practical application first.17:24–22:57 · Guest disagreement 1/10 Host Q&A and Event Conclusion Host Matt Turck opens Q&A by concisely summarizing DataRobot's automated model selection value proposition and inviting Achin to explain the product. Achin and the audience discuss Kaggle performance and unexpected algorithmic results.0:48–4:26 · Matt pushing back 0/10 Jeremy Achin Background and DataRobot Company Overview Achin provides his background as an insurance data scientist and Kaggle competitor while introducing DataRobot and citing Leo Breiman's 2001 paper on statistical modeling. Because this is a keynote monologue, the host does not participate.4:26–6:34 · Matt pushing back 0/10 Redefining Black Boxes and Model Interpretability Achin aggressively redefines 'black box' as a term used by people afraid of technology who want to maintain 20-year-old habits. He outlines technical criteria for model transparency that apply across complex ensemble algorithms.6:34–10:25 · Matt pushing back 0/10 Case Study: Hospital Readmission and Partial Dependence Achin presents a hospital readmission case study to demonstrate how partial dependence plots reveal marginal non-linear feature effects in complex models. He explains how this approach provides model interpretability without relying on simple linear regression.10:25–14:03 · Matt pushing back 0/10 Critiquing Compliance and Arbitrary Representations of Reality Achin heavily critiques traditional statisticians and compliance teams, defining compliance as a last resort excuse to defend linear regression status quo. He uses humorous parables like statisticians in a bar and killer potatoes to show how regression coefficients can create arbitrary representations of reality.14:03–17:24 · Matt pushing back 0/10 Democratizing Data Science and Presentation Takeaways Achin critiques traditional data science education pipelines for dropping students before teaching practical application. He argues that automated modeling tools will solve the data scientist shortage by enabling practical application first.17:24–22:57 · Matt pushing back 2/10 Host Q&A and Event Conclusion Host Matt Turck opens Q&A by concisely summarizing DataRobot's automated model selection value proposition and inviting Achin to explain the product. Achin and the audience discuss Kaggle performance and unexpected algorithmic results.

speaking balance: gold is Matt, purple is the guest (3 minute bins)

0:00 · Matt 0% · guest 100%0:00 · Matt 0% · guest 100%3:00 · Matt 0% · guest 100%3:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%6:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%9:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%12:00 · Matt 0% · guest 100%15:00 · Matt 14.9% · guest 85.1%15:00 · Matt 14.9% · guest 85.1%18:00 · Matt 2.3% · guest 97.7%18:00 · Matt 2.3% · guest 97.7%21:00 · Matt 29.3% · guest 70.7%21:00 · Matt 29.3% · guest 70.7%
Sharpest disagreement ▶ 10:55 Redefining Compliance as Defending Status Quo

Achin explicitly mocks standard industry reliance on linear regression, defining compliance as a desperate attempt to defend outdated habits while acknowledging he may offend audience members.

Hardest push from Matt ▶ 17:30 Prompting Product Explanation

Host Matt Turck intervenes post-presentation to gently push Achin into explaining DataRobot's actual commercial product after Achin intentionally avoided a sales pitch.

Biggest teaching moment ▶ 8:15 Explaining Partial Dependence Plots

Achin educates the room on how partial dependence curves isolate non-linear feature relationships inside black-box models, demonstrating how modern machine learning achieves interpretability without linear assumptions.

Matt holds his own ▶ 17:30 Summarizing Automated Model Selection

Host Matt Turck demonstrates his domain knowledge by accurately summarizing how DataRobot automates testing multiple algorithms against arbitrary dataset types.

the scores for every segment, with the reasoning behind each
ChapterTopicMatt as informed peerGuest teachingGuest disagreementMatt pushing backWhy
Jeremy Achin Background and DataRobot Company Overview 0210 Achin provides his background as an insurance data scientist and Kaggle competitor while introducing DataRobot and citing Leo Breiman's 2001 paper on statistical modeling. Because this is a keynote monologue, the host does not participate.
Redefining Black Boxes and Model Interpretability 0440 Achin aggressively redefines 'black box' as a term used by people afraid of technology who want to maintain 20-year-old habits. He outlines technical criteria for model transparency that apply across complex ensemble algorithms.
Case Study: Hospital Readmission and Partial Dependence 0510 Achin presents a hospital readmission case study to demonstrate how partial dependence plots reveal marginal non-linear feature effects in complex models. He explains how this approach provides model interpretability without relying on simple linear regression.
Critiquing Compliance and Arbitrary Representations of Reality 0670 Achin heavily critiques traditional statisticians and compliance teams, defining compliance as a last resort excuse to defend linear regression status quo. He uses humorous parables like statisticians in a bar and killer potatoes to show how regression coefficients can create arbitrary representations of reality.
Democratizing Data Science and Presentation Takeaways 0430 Achin critiques traditional data science education pipelines for dropping students before teaching practical application. He argues that automated modeling tools will solve the data scientist shortage by enabling practical application first.
Host Q&A and Event Conclusion 3312 Host Matt Turck opens Q&A by concisely summarizing DataRobot's automated model selection value proposition and inviting Achin to explain the product. Achin and the audience discuss Kaggle performance and unexpected algorithmic results.

Statements from this episode (9)

Assertion Supported
Achin: DataRobot employees held Kaggle's top rank almost continuously for three years
“Basically, over the last three years, somebody from our company has been number, ranked number one almost the entire time.”
Jeremy Achin Nov 23, 2015 ▶ 2:03
Assertion Not checkable as stated
Achin: Large companies still employ armies of people manually building regression models
“But there's still armies of people mostly in big companies you know, building regression models manually, taking months to do that.”
Jeremy Achin Nov 23, 2015 ▶ 3:56
Insight
Achin: Complex machine learning models can be interpreted using modern techniques
“I would argue, though, that even very complicated models can be interpreted using modern techniques.”
Jeremy Achin Nov 23, 2015 ▶ 4:19
Opinion
Achin: 'Black box' label reflects fear of new technology
“It's a phrase people use when they're scared of technology they don't understand and want to keep doing the same thing they've been doing for the last 20 years.”
Jeremy Achin Nov 23, 2015 ▶ 4:37
Disclosure
Achin penalized job candidates for choosing the wrong modeling textbook
“I love this book so much that I would use it in interviews, so I, when somebody would come in for an interview, I would ask them, if you had to go on a desert island And do modeling consulting for the rest of your life, and you can bring one book, what would i…”
Jeremy Achin Nov 23, 2015 ▶ 10:00
Opinion
Achin: Compliance is used to defend outdated, inaccurate regression models
“A word people use as a last resort to defend the status quo after they realize that their hundred variable regression model is an arbitrary representation of reality that is less accurate, robust, and interpretable than the modern, ah, alternatives.”
Jeremy Achin Nov 23, 2015 ▶ 11:03
Prediction Not checkable as stated
Achin: Automated tools and education will solve the data scientist shortage
“And, ah, the shortage of data scientists will be solved by a combination of pragmatic education, practical education, and, ah, tools that automate the modeling process and automate data science to levels which are, right now, probably most of us in the room th…”
Jeremy Achin Nov 23, 2015 ▶ 17:02
Assertion Contradicted
Achin: Non-data scientist advisor placed 5th in Kaggle competition using DataRobot
“One of our advisors, who's not a data scientist, knows nothing about insurance he came in fifth place. And this is out of six hundred-something people including thirty-something Liberty Mutual employees that have competed.”
Jeremy Achin Nov 23, 2015 ▶ 19:55
Insight
Achin: Gradient boosted trees perform best over 40% of the time
“Gradient boosted trees tends to be you know, the, one of the top algorithms, like 40 plus percent of the time.”
Jeremy Achin Nov 23, 2015 ▶ 21:27
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 400 conversations transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.