Andreessen: AI models keep improving when trained far past expected limits
Marc Andreessen · Startup Building: Challenges & Opportunities · May 2, 2024 · at 47:58
Marc Andreessen, co-founder of Andreessen Horowitz (a16z), discusses recent findings in LLM training scaling, referencing Meta's Llama model experiments.
0:00 / 0:31exact quote · 31.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“There's been this theory for a long time that you basically take, you take input into train and AI, and you get the, have the chips all set up and everything, and then you train it up to a point, you let the training process run for a certain amount of time, and then you hit a point of diminishing returns, and at some point you stop training and release it and you just, at some point you just can't make the model smarter by training it more, and it's just, it's turning out, there's more and more cases now where people are realizing, no, actually, if you keep training it, and actually, if you keep training it to what people used to think were kind of, Absurd extents. Like if you train it to be like a hundred, you know, train it for a hundred times longer than the sort of predictions would have suggested.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Marc Andreessen
Andreessen: US institutions recruit African immigrants to avoid hiring African Americans
“Both the universities and the corporate employers are literally, this is true, they're literally, they literally import Africans so that they don't have to hire African Americans because they still count as black.”
Andreessen: Big Tech is aggressively lobbying for regulatory capture against startups
“Quite honestly, I mean, part of what's happening is big tech is against us. There, there's a, there's basically a lot of the political issues now are a civil war inside the tech industry where, The big tech companies are lobbying very aggressively for what the…”
Andreessen: Vinod Khosla lobbied Congress to ban open-source AI
“Vinod has been lobbying to ban open source AI. And by the way, he denies that he's been doing this, but I saw him with my own eyes when the US Congressional China Committee came to Stanford I was in the meeting where he was with, you know, 30 20 or 30 congress…”
Andreessen: Automated and AI-augmented warfare will make conflict much safer
“I actually think warfare is going to get a lot safer. Like, I think, actually, automated warfare would be much, much safer.”
Andreessen: Biden officials explicitly told a16z they planned to ban open-source AI
“They told us that explicitly. And yeah. And they told us exactly what you just said. They told us they're gonna ban open source. And when we challenged them on the ability to ban open source, because it's, you know, we're talking about, you know, math, you kno…”
Andreessen: Paul Thomas Anderson's new film is 'Antifa the movie'
“Arguably, it is kind of salient to our times, because I think if you were to retitle the movie, you would just title it Antifa the movie. Like, it's, like, it's like a full-throated celebration of basically violent social change, violent social revolution, lik…”