Prakash: Together AI builds cloud footprint avoiding major hyperscalers
Vipul Ved Prakash · Building an open AI company - with Ce and Vipul of Together AI · Feb 8, 2024 · at 2:06
Vipul Ved Prakash, co-founder and CEO of Together AI, explains Together AI's infrastructure strategy of aggregating distributed data centers instead of relying on major hyperscaler cloud providers.
0:00 / 0:20exact quote · 20.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The way we structure our cloud is by combining data centers around the world instead of you know we are today not located in hyperscalers. We have built a footprint of you know, AI supercomputers in this sort of a disaggregated, decentralized manner.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Vipul Ved Prakash
Prakash: Foundation models lack data moats and are constrained by capital
“There aren't really big data moats around foundation models. They are built from a subset of the web. What is difficult is the cost of capital to build these, and what, one of the ways in which you can reduce this cost is by making more efficient systems.”
Prakash: Transformer architectures will not reach 5,000 tokens per second inference
“We are running into you know, the limits of how fast you can make transformers and you know, we want inference at 5000 tokens per second. And I don't think we will get there with transformers”
Prakash predicts up to 5 million AI GPUs will sell in 2024
“There is four to five million GPUs that will be sold this year. NVIDIA and others.”
Prakash: SemiAnalysis inference report erred by assuming unpriced input tokens
“I think there were some errors in that analysis. In particular we were trying to decode it, and one of the things we noticed is that it assumed that input tokens weren't being priced. So I think that may have been an error in the model.”
Prakash: AnyScale LLM benchmarks conflicted with internal provider testing
“Everyone sort of had a reaction to it because it just didn't match their benchmarks that we've all run internally against different services.”
Prakash: AI discourse is overly dominated by sci-fi-informed doomerism
“I think we have had this Very you know, sort of a doomerism view of it really kind of informed by science fiction, you know, dystopian science fiction and Terminator, and I don't think we have a kind of a positive or a realistic, really framework coming from, …”