Conover: Databricks' Dolly model was never state-of-the-art, but proved a concept
Mike Conover · How AI is Eating Finance - with Mike Conover of Brightwave · Jun 11, 2024 · at 1:03:33
Mike Conover, creator of Databricks' Dolly open-source LLM, clarifies the model's original capabilities and performance standing.
0:00 / 0:09exact quote · 9.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“DALI was never the best open source model, but it demonstrated a new, Something that was not obvious to many people at the time. Yeah, but we always were clear that it was never state of the art.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mike Conover
Conover: AI models outclass human working memory, not core reasoning
“It's not clear that you get superhuman reasoning capabilities from human level demonstrations of skill. And by that, I mean the pre-training corpus, but then additionally, the fine tuning corpuses, I think you largely mimic the demonstrations that are present …”
Conover: Million-token context windows fail to extract deep insights from SEC filings
“It, I think empirically is not the case that you can just throw all of the SEC filings in, you know, a million token context window and get deep insight that is useful out the other end.”
Conover: Push-button automated financial modeling misses the core purpose of modeling
“The other piece of this is that the financial modeling is often very, when we talk to our users, it's very personal. So they have a specific view of how a company is structured. They have the, you know, one key driver of asset performance that they think is re…”
Conover: Investment thesis idea generation is absolutely automatable with AI
“I think that process of idea generation is absolutely automatable.”
Conover: AI model developers are absolutely overfitting to public evaluation benchmarks
“And I think the work around over, you know, overfitting on the test, I think is like that. 100% is happening.”
Conover: Economic incentives to pre-train commodity foundation models from scratch are diminishing
“The incentives, the economic incentives for companies to train their own foundation models, I think, are diminishing. So the, like, window in which you are the dominant pre-train, and let's say that you spend five to forty million dollars, you know, for like a…”