Opinion
Ghodsi: All current LLM evaluation benchmarks are completely bullshit
“I kind of think all the benchmarks are bullshit, and so all these, so all the LLM benchmarks, here's how it works.”
Opinion
Ghodsi: Product-led growth is overrated for enterprise software
“This PLG thing is just overrated for the kind of business we have in the kind of market we are in.”
Insight
Horowitz warned Ghodsi that selling Databricks early would cause lifelong regret
“Hey, you can do whatever you want. You can sell. You're going to make a lot of money and you'll be super successful in life. But you know, if you're like me, you're going to look back the rest of your life thinking, you know, I missed that one shot. That was t…”
Insight
Ghodsi: A commercial open-source company's biggest enemy is its free project
“But the problem was that you know, as it is often with open sources that you just everyone's just downloading the open source version. Actually your biggest enemy is your open source project.”
Insight
Ghodsi: Founders fail by hiring their own archetype for unfamiliar roles
“They'll do well in their own archetype. When they have to step outside of their own archetype, they make a mistake. They hire people that are like their own archetype in other roles where that could be lethal.”
Disclosure
Ghodsi: Databricks passed on acquiring a successful company over weak talent
“There is one really good company that well, one really successful company that we never acquired. And I always vetoed it whenever it came up because I just think that the quality of their employee base is not great. And I didn't want it to dilute Databricks.”
Assertion Not checkable as stated
Ghodsi: Reports of $100M AI compensation packages are largely false
“I don't think anyone's getting a hundred million dollar offers. You know, I mean, yes, there is like, One, you know, character that AI and so on, but I didn't think it's actually true.”
Assertion Not checkable as stated
Ghodsi: Product-led growth was one of Databricks' biggest failures
“PLG is something that we had tried very hard and it didn't work for us. Like, and actually one of our biggest failures was PLG at Databricks.”
Assertion Not checkable as stated
Ghodsi: Enterprise CEOs want proprietary AI models to protect their data
“One of the things that's really interesting that's happened in the sort of brains of the CEOs and the boards is that they realize Maybe I can beat my competition. Maybe this is the kryptonite that will help me kill my enemy. I have the data with generative AI.…”
Insight
Ghodsi: Smaller custom models can beat large LLMs on domain accuracy
“And there you're better off if you have a good data set to train, you can train a smaller model. The latency will be faster to use it later, and it will be cheaper to use it later, and yes, you can have absolutely accuracy that beats the really large model, bu…”
Disclosure
Ghodsi: Databricks restricted MosaicML sales because of severe GPU shortages
“So we bought Mosaic. I did not unleash our sales force and go to market of 3000 people. To sell the thing that we bought because we just can't satisfy the demand. Like there's not enough GPUs.”
Insight
Ghodsi: Hype around the largest LLMs mirrors 2000s focus on Cisco
“So I think it's a little bit like right now like that. Who has the largest LLM? Obviously whoever can build the largest one that can train it the most obviously will own all of AI and all the future of humanity. But just like the internet, someone will show up…”
What-if
Ghodsi: AI progress would be far behind without Meta releasing Llama
“If the original Llama was never released, What would the state of the world and our view of AI be right now? We would be way further behind, right? And A, it was a big model you know, by what existed in open source and it was open sourced. And both of those th…”
Assertion Supported
Ghodsi: MMLU benchmark scores are inflated due to web data contamination
“MMLU is just a multi-choice question that's on the web. Ask a question. Here's, is the answer A, B, C, D, and then it says what the right answer is. And it's on the web. You can deliberately train on it and create an LLM that crushes it on that. Okay. Or you c…”
Prediction Not checkable as stated
Ghodsi: Humanity is very far away from autonomous AI self-reproduction
“Once you have reproduction and, you know, the building of new ones automatically once you crack the code on that loop, yes, then I think we're fucked, but we're very far away from that.”
Opinion
Ghodsi: Founders who mis-hire executives owe them maximum offboarding generosity
“I disagree with that. I think from then on, it's you owe them. You're the one that made the mistake of hiring them to be as generous as possible within reason.”
Insight
Ghodsi: Most startup BD deals fail because startups offer no commensurate value
“So like in, in these BD deals there always has to be a give and get that actually is kind of commensurate. And this is why most of these deals fall apart and they don't work. There has to be something that you as a small player can give that they don't have. U…”
Insight
Ghodsi: Equity dilution budget equals market cap divided by employee count
“The way to think about it is your market cap divided by number of employees. That's how much money you can give away in terms of dilution.”
Insight
Ghodsi: A CEO questioning an executive's fit means they must be replaced
“It's when you're asking other people, what do you think? Or when you're starting to have the thoughts of like, is this the right person or is it not the right person? You're done.”
Assertion Not checkable as stated
Ghodsi: Early product-led growth capped Databricks deal sizes at $30K
“And we could not do a deal that was bigger than 20 30 K. It was just impossible.”
Insight
Ghodsi: Framing feedback as optional career coaching increases employee receptivity
“I think a lot of feedback can be recast. Into, you know, I'm just here to help you, but feel free to completely ignore this advice. But, you know, if you want to be really successful, if you want to get that job, or if you want to get that project next time, i…”
Opinion
Ghodsi highly recommends competitor Frank Slootman's book Amp It Up
“One of my competitors, Frank Slootman wrote the book called Amp It Up. It's a great book on how you get you know, execution into a company, like getting a high performance culture read. Everybody's always trying to excel and do better and better sort of that k…”
Insight
Ghodsi: Every term negotiated in M&A sets a precedent for future deals
“Every little thing you agree to there is a precedent for the next deal.”
Insight
Ghodsi: Ideal hires combine big-tech scale experience with startup grit
“I love to hire people who have seen great at a big company, like, or I don't know if it's great, but they've seen process scale, big company. They've been at a Google, they've been at Amazon, they understand the processes so that I understand how to navigate a…”
Ali Ghodsi