Conover: Repeatable AI annotation pipelines require unsexy people management
Mike Conover · How AI is Eating Finance - with Mike Conover of Brightwave · Jun 11, 2024 · at 30:14
Brightwave founder Mike Conover explains why enterprises struggle to replicate employee-led data labeling initiatives like Databricks' Dolly-15k.
0:00 / 0:16exact quote · 16.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It's one thing to do like a single monolithic push to create a, Training data set like that, or an evaluation corpus, but I think it's another to have a repeatable process, and a lot of that, I think, realistically is pretty unsexy, like, people management work.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mike Conover
Conover: AI models outclass human working memory, not core reasoning
“It's not clear that you get superhuman reasoning capabilities from human level demonstrations of skill. And by that, I mean the pre-training corpus, but then additionally, the fine tuning corpuses, I think you largely mimic the demonstrations that are present …”
Conover: Million-token context windows fail to extract deep insights from SEC filings
“It, I think empirically is not the case that you can just throw all of the SEC filings in, you know, a million token context window and get deep insight that is useful out the other end.”
Conover: Push-button automated financial modeling misses the core purpose of modeling
“The other piece of this is that the financial modeling is often very, when we talk to our users, it's very personal. So they have a specific view of how a company is structured. They have the, you know, one key driver of asset performance that they think is re…”
Conover: Investment thesis idea generation is absolutely automatable with AI
“I think that process of idea generation is absolutely automatable.”
Conover: AI model developers are absolutely overfitting to public evaluation benchmarks
“And I think the work around over, you know, overfitting on the test, I think is like that. 100% is happening.”
Conover: Economic incentives to pre-train commodity foundation models from scratch are diminishing
“The incentives, the economic incentives for companies to train their own foundation models, I think, are diminishing. So the, like, window in which you are the dominant pre-train, and let's say that you spend five to forty million dollars, you know, for like a…”