May 31, 2023 · 22m · mad
Entering the Data-Centric Era of Foundation Models with Alex Ratner, Co-Founder & CEO of Snorkel AI
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
At Data Driven NYC, Snorkel AI CEO Alex Ratner demonstrates why private enterprise data and programmatic data workflows are essential for customizing foundation models into accurate, production-ready AI systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
Matt as informed peer 0.0 Guest teaching 0.4 Guest disagreement 0.6 Matt pushing back 0.0
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 5:35 Critique of OpenAI corporate lobbying
Alex sarcastically highlights how open-source model replication challenges closed API providers who are lobbying in Congress to restrict open-source AI.
Hardest push from Matt ▶ 20:11 Audience challenge on model bias loopsAn audience member intervenes during Q&A to question whether programmatic data labeling creates a circular feedback loop that locks models into restrictive norms.
Biggest teaching moment ▶ 20:40 Clarifying programmatic labeling generalizationAlex educates the audience member on machine learning generalization, explaining how labeling subsets of data via weak supervision avoids circular model feedback.
Matt holds his own ▶ 22:10 Brief host wrap-upThe host Matt Turck does not participate in technical discussion, providing only a brief closing sentence at the conclusion of the event.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Welcome and Presentation Overview on Data-Centric AI | 0 | 0 | 0 | 0 | Alex Ratner introduces the talk on data-centric AI and foundation models. As a monologue presentation, the host does not speak or engage during this segment. | |
| Key Point 1: Private Enterprise Data as the AI Moat | 0 | 0 | 2 | 0 | Alex argues enterprise data is the only durable moat in AI and takes a brief jab at OpenAI lobbying Congress. The segment is a solo lecture with no host participation. | |
| Key Point 2: Adapting Foundation Models for High-Accuracy Tasks | 0 | 0 | 0 | 0 | Alex presents evidence showing why foundation models require data-centric tuning for high accuracy in enterprise tasks. The monologue presentation continues without host involvement. | |
| Programmatic Data-Centric AI and the Snorkel Flow Platform | 0 | 0 | 0 | 0 | Alex explains weak supervision and programmatic labeling functions on the Snorkel platform. Host metrics remain at zero due to lack of host speech. | |
| Key Point 3: Distilling Foundation Models into Specialist Models | 0 | 2 | 1 | 0 | An audience member asks a technical question about feedback loops in automated labeling, which Alex addresses before host Matt Turck briefly closes the presentation. |
Statements from this episode (10)
Ratner: Foundation models are incomplete solutions for real-world enterprise AI
“In most real-world use cases and we'll get into how I define that in a bit, there's still a last mile to be traversed. So it's not that they're overhyped, per se, they're just not the complete solution.”
Ratner: Foundation models will require customization on enterprise-specific data
“Most of these foundation models, like a GPT-IV, five, six, seven, are going to need to be customized on your specific data and knowledge and workloads.”
Alex Ratner: Enterprise data is AI's only durable moat
“Enterprise data knowledge is the durable moat in AI, I'd argue the only durable one”
Ratner: $200 of ChatGPT calls can clone closed models into open ones
“If you take a couple hundred bucks of API calls to, say, ChatGPT, and you graph that onto a model that is substantially smaller, say a seven billion parameter model like LLAMA, or now increasingly fully open for commercial use ones, like Red Pajama is one that…”
Ratner: Private data models will exceed closed models in specialized tasks
“Closed source models, like a GPT-IV, five, six, seven, whatever comes, are going to be very hard to match in terms of generalist capability for, say, consumer use cases that are reflected in the web data they're trained on and the flywheels that get powered by…”
Ratner: BloombergGPT achieves only low 60s accuracy on specialized financial tasks
“If you actually open the first page of the Bloomberg GPT paper, the FinServe tasks that it does better on by using private financial data are still getting low sixties in terms of accuracy.”
Alex Ratner: Traditional model-centric AI development is dying off
“Tweaking the model, finding a fancier algorithm, all this kind of traditional model-centric development that, frankly, we still mostly teach in data science one-on-one courses as what data science is, is effectively dying off.”
Ratner: DataComp benchmark beat OpenAI models purely through data curation
“Just by cleaning, curating, sampling, filtering the data, we get a new state of the art score at compute parity, beating open AI models and others.”
Ratner: Distillation creates specialized models 100,000x smaller with higher accuracy
“It shows that you can take these massive kind of generalist foundation models, And not only can you tune them using that kind of programmatic labeling to be more accurate on a given task that you need to be really accurate on, but you can then distill down ver…”
Ratner: Base models will open-source while production uses specialized models
“There are going to be these base models. They're probably going to be increasingly open source. And then the reality of what actually ships in production is going to be a whole kind of family tree of smaller specialized models that are tuned and adapted via da…”