Conrad: Test-time inference will significantly expand inference compute demand
Evan Conrad · SF Compute: Commoditizing Compute · Apr 11, 2025 · at 31:37
Evan Conrad, co-founder of SF Compute, explains how test-time compute reasoning models shift inference infrastructure requirements.
0:00 / 0:09exact quote · 9.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The thing I do feel reasonably confident about saying is that the test time inference is probably going to quite significantly expand the amount of compute that was used for inference.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Evan Conrad
Conrad: Hyperscalers will probably lose significant money reselling Nvidia GPUs
“My intuition is that the hyperscalers are probably going to lose a lot of money, and they know they're going to lose a lot of money on reselling NVIDIA GPUs at least.”
Conrad: Decentralized compute networks will never beat co-located InfiniBand clusters
“I just don't really think this is gonna ever be more efficient than a fully interconnected cluster with Infiniband, or, you know, whatever sort of next spec might be. Like, I could be completely wrong, but Speedolite is really hard to beat. And regardless of w…”
Conrad: The AI VC bubble will pop and fail to return capital
“So what you've done by not having a future is you've inflated the venture capital market. And that is a bubble that's totally going to pop at some point. Like a lot of the companies are not going to work. And the valuations are not going to work. And what's go…”
Conrad: Software margins on GPU clusters drive customers to build in-house
“So if you have a 10% margin increase because you have great software on your billion dollars, the customers are that price sensitive. They will immediately switch off if they can, because why wouldn't you? You would just take that hundred million dollars, you'…”
Conrad: CoreWeave's debt-financed long-term contract model is optimal for GPUs
“So that means that the best way to make money in GPUs was to do basically exactly what CoreWeave did which is go out and sign only long-term contracts, pretty much ignore the bottom end of the market completely, and then maximize your long-term contracts with …”
Conrad: GPU businesses succeed as pure real estate or pure software, not both
“The GPU clouds are fantastic real estate businesses. If you treat them like real estate businesses, you will make a lot of money. The, Cloud services you can make on that, all the software you want to make on that, you can do that fantastically. If you don't o…”