Ratner: BloombergGPT achieves only low 60s accuracy on specialized financial tasks
Alex Ratner · Entering the Data-Centric Era of Foundation Models with Alex Ratner, Co-Founder & CEO of Snorkel AI · May 31, 2023 · at 9:31
Alex Ratner, CEO of Snorkel AI, references results from the BloombergGPT research paper during a presentation on adapting foundation models for enterprise tasks.
0:00 / 0:10exact quote · 10.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“If you actually open the first page of the Bloomberg GPT paper, the FinServe tasks that it does better on by using private financial data are still getting low sixties in terms of accuracy.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Alex Ratner
Alex Ratner: Traditional model-centric AI development is dying off
“Tweaking the model, finding a fancier algorithm, all this kind of traditional model-centric development that, frankly, we still mostly teach in data science one-on-one courses as what data science is, is effectively dying off.”
Alex Ratner: Enterprise data is AI's only durable moat
“Enterprise data knowledge is the durable moat in AI, I'd argue the only durable one”
Ratner: $200 of ChatGPT calls can clone closed models into open ones
“If you take a couple hundred bucks of API calls to, say, ChatGPT, and you graph that onto a model that is substantially smaller, say a seven billion parameter model like LLAMA, or now increasingly fully open for commercial use ones, like Red Pajama is one that…”
Ratner: Private data models will exceed closed models in specialized tasks
“Closed source models, like a GPT-IV, five, six, seven, whatever comes, are going to be very hard to match in terms of generalist capability for, say, consumer use cases that are reflected in the web data they're trained on and the flywheels that get powered by…”
Ratner: Distillation creates specialized models 100,000x smaller with higher accuracy
“It shows that you can take these massive kind of generalist foundation models, And not only can you tune them using that kind of programmatic labeling to be more accurate on a given task that you need to be really accurate on, but you can then distill down ver…”
Ratner: Base models will open-source while production uses specialized models
“There are going to be these base models. They're probably going to be increasingly open source. And then the reality of what actually ships in production is going to be a whole kind of family tree of smaller specialized models that are tuned and adapted via da…”