Ratner: Private data models will exceed closed models in specialized tasks
Alex Ratner · Entering the Data-Centric Era of Foundation Models with Alex Ratner, Co-Founder & CEO of Snorkel AI · May 31, 2023 · at 8:11
Alex Ratner, CEO of Snorkel AI, contrasts generalist consumer LLMs with specialized enterprise models built on proprietary data.
0:00 / 0:22exact quote · 22.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Closed source models, like a GPT-IV, five, six, seven, whatever comes, are going to be very hard to match in terms of generalist capability for, say, consumer use cases that are reflected in the web data they're trained on and the flywheels that get powered by this mass consumer adoption. But for any kind of specialized use case, they're going to be very readily matched and exceeded when you start using private data and knowledge.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Alex Ratner
Alex Ratner: Traditional model-centric AI development is dying off
“Tweaking the model, finding a fancier algorithm, all this kind of traditional model-centric development that, frankly, we still mostly teach in data science one-on-one courses as what data science is, is effectively dying off.”
Alex Ratner: Enterprise data is AI's only durable moat
“Enterprise data knowledge is the durable moat in AI, I'd argue the only durable one”
Ratner: $200 of ChatGPT calls can clone closed models into open ones
“If you take a couple hundred bucks of API calls to, say, ChatGPT, and you graph that onto a model that is substantially smaller, say a seven billion parameter model like LLAMA, or now increasingly fully open for commercial use ones, like Red Pajama is one that…”
Ratner: Distillation creates specialized models 100,000x smaller with higher accuracy
“It shows that you can take these massive kind of generalist foundation models, And not only can you tune them using that kind of programmatic labeling to be more accurate on a given task that you need to be really accurate on, but you can then distill down ver…”
Ratner: Base models will open-source while production uses specialized models
“There are going to be these base models. They're probably going to be increasingly open source. And then the reality of what actually ships in production is going to be a whole kind of family tree of smaller specialized models that are tuned and adapted via da…”