Polosukhin: Decentralized AI training is unrealistic due to GPU bandwidth requirements
Illia Polosukhin · No Priors Ep. 32 | With NEAR’s Illia Polosukhin · Sep 14, 2023 · at 22:55
Transformer co-author Illia Polosukhin evaluates the technical feasibility of training foundation AI models across distributed, blockchain-based GPU networks.
0:00 / 0:27exact quote · 27.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And the reality right now that the requirements on bandwidth, right? Like people who are training these models right now, they have like a, you know, 800 gigabit connect right between the GPUs, right? So Maybe you have a hundred megabits on between this, usually not, and you need to like replay and like work around problems for decentralized. So I think decentralized training right now is like still not as realistic, although there's some research people are trying.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Illia Polosukhin
Polosukhin: AI agent CEOs will run organizations and replace middle management
“I really think one of the most interesting cases is organizations that are run completely by AI, right? Where quote unquote CEO role is taken by AI agent. Who's tasked by, you know, by community or board of directors or whatever is oversight governance is to, …”
Polosukhin: Web3 and AI will start replacing traditional SaaS software
“So I think a lot of between web three and AI, a lot of SaaS will actually start being replaced because right now what SaaS is, is like one database. With a specific UI for a specific problem.”
Polosukhin: Transformers will be very hard for alternative architectures to match
“The simplicity of this architecture and like, indeed, like the amount of optimization that's going into this right now is just, it will be really hard to match”
Polosukhin: First organizations run by AI agents will emerge in 2023
“I think we'll see, you know, first organizations like this, probably even this year where Potentially with a simpler mission, some kind of more straightforward like KPI metrics, but where kind of this information propagation and onboarding of people happens al…”
Polosukhin: DAOs will be the first place AI managerial agents emerge
“I think DAOs is, and especially what happened with DAOs, there was a lot of people who were really excited about DAOs kind of as a concept. And so they put a lot of time running them, but it's actually a very like not interesting job, right? It's like you onbo…”
Polosukhin: Society needs human alignment rather than AI alignment
“So I have this view that we need human alignment instead of AI alignment. So right now, kind of when we talk about, you know, hey, we need to align AIs with like human values, but the reality is that, you know, all the problems that exist, they all exist becau…”