The Ledger, every show

“The whole AGI effective altruist this whole ecosystem, it never resonated with me. It felt like cause playing. It felt like people were LARPing a new religion and all of this stuff like create God.”

“Anyone who's selling you doom and gloom, I think is wrong.”

“We will build AGI if what you mean is very useful, generally capable technology that can do a lot of the stuff that humans can do and flex into a lot of different domains. If what you mean is, you know, are we gonna build God? No.”

“I don't think that models are actually getting commoditized. I think what you see is you see price dumping. And so you see people giving it out for free, giving it out at a loss, giving out at zero margin. And so they see the prices coming down and they assume…”

“Well, because I think that we will never see mass unemployment of humans. I think that this technology is going to unlock more opportunities. It will let us do more as opposed to Scaling back what we do.”

“As more and more evidence emerges that these models are much more controllable than we may have thought that they're a little bit less capable than we may have thought it's harder and harder to make that narrative.”

“Not happening. It's not happening. It improves for, it can self-improve for a while, and then it tapers off. And so, yeah, you get some good improvement out of it, which is why we use it, but then it plateaus. It doesn't just keep going forever.”

“I actually take, I think the less popular view, which is The internet is enough, and by observation, you can actually learn enough to be extremely, extremely compelling. I think that's if we're talking about AGI and doing things as well as humans do, I think t…”

“It's really dangerous when you make yourself a subsidiary of your cloud provider.”

“At least like lately, or in the new OpenAI they're like a product company. They're like hardcore building a consumer product. That is their objective, and it's working. People love that product. It's, you know, a household name at this point. So I think in the…”

“If you're not able to actually transform the model to be better at the thing that you care about, if you're not the one building the model, if you're just a consumer of the model you're structurally disadvantaged to build that product.”

“The fears around displacement and replacement, both on the consumer side, where we're all gonna get addicted to chatting to these chatbots, and on the workplace, the end of work, you know, there's gonna be mass unemployment. I can't see that happening.”

“In-person is just so much better. It's just, you can't even quantify the productivity lift from in-person work.”

“And so one of the big shocks is how over the past eight years, how little things have changed. Like it, it's really surprising to me. That the Transformers we train today looks so similar to what was back then.”

“To say they didn't lean hard enough into language modeling, like just pure Sequence modeling of text on the internet. That's, I think the accurate statement. That's what OpenAI did early and uniquely well.”

“Now there are these discrete diffusion models, which do diffusion, which has been super popular for, like, image understanding, image generation. It's doing that same process for language models, but I still don't see that replacing the transformer.”

“It's easy to create a reasoning model. It's dramatically cheaper than pre-training. And so it's accessible. And so there's this huge intelligence uplift that comes for really quite little effort.”

“Is it better than the world's best doctor at prescribing drugs? Probably not. Will it get there? Probably.”

“Synthetic data is incredibly effective. It's now the majority of the data that we train on for creating something like command A.”

“So we can take something that used to be a month. And bring it down to, you know, four hours, eight hours.”

“We're not going to build a ChatGPT competitor. What we want to build is a platform and a series of products to enable enterprises to adopt this technology and make it valuable.”

“Even if we didn't train a single new language model, like, okay, all the data centers blow up. We can't improve the LLM. We only have what we have today. There's a half decade of work to go integrate this into the economy, to build all these things, to build t…”

“What we've seen is today you can build a model that's as good as GPT-IV in all the things that enterprises might care about. For ten million dollars, twenty million dollars, like just orders of magnitude less than what was spent to develop that model. And so i…”

“So that's the strategy is don't lead. Don't burn, you know, three, five, seven billion dollars a year to be at the front, be six months behind. And offer something to market to enterprises that actually fits their needs at a price point that makes sense for th…”

“We're starting to enter into a sort of flat part of the curve and we're certainly past the point where if you just interact with a model, You can know how smart it is. Like the vibe checks, they're losing utility.”

“I don't need to go double the size of my supercomputer to hit a requisite intelligence threshold. I can just double the amount of inference time compute that my customers pay for.”

“At some stage we're gonna have to give these models the ability to run their own experiments to fill in areas of their knowledge that they're curious about. But I think that's quite Quite a ways away. And it's going to be tough to scale that. It will take many…”

“I think in reality, the state of the world is there's a total technological refactor that's going on right now and will last the next 10 to 15 years. And it's kind of like we have to repave every road on the planet. And there's like four or five companies that…”

“I certainly understand the urge for people to see the numbers being spent on training and be concerned that it's not going to recoup in value, but I think that Those numbers are actually small relative to the long-term value that the technology will deliver.”

“I don't know about God. Like, I don't think I'd ever use that term to describe what's coming. I think we're going to have some really powerful and useful tools. Emerge. I think that's what's coming. The idea that we're building AGI or something that's just gon…”

“I don't think within any achievable scaling up for humanity that will reach that tipping point. It just saturates the gains become much, much smaller. And so you're much less willing to want to pay double the price for a minute difference. But it is pretty con…”

“I did not sign that letter.”

“I think they can interpolate between skills, and so if they've seen how to do A, and they've seen how to do B, they can get kind of the average of A and B but they don't just go completely beyond anything that they've seen.”

“Synthetic data Probably doesn't get us out of that, that issue. I actually, I don't know if synthetic data outside of easily verifiable domains like math, it's hard to use synthetic data to drive outcomes.”

“I am not aware of it. I don't think I have any example of that happening. It's very assistive actually. So it's less about replacement. It's more about augmentation. Like at the moment, what everyone's building are tools to augment their workforce to make them…”

“The reality of the matter is there's no market for last year's model.”

“I think we'll continuously exist in a world of multiple models, some focused and verticalized, others completely horizontal.”

“What's funny is that curriculum learning has actually failed in machine learning. We don't really do curriculum learning. It's just throw the hardest material and the easiest material all at the same time and let the model figure it out.”

“GPT-IV, if it's true what they say, and it's 1.7 trillion parameters, this big MOE, we have models that are better than that model that are like, thirteen billion parameters.”

“Yeah, I mean, I think it certainly requires exponential input. You know, you need to continuously be doubling your compute in order to sustain linear gains in, in intelligence. But I think that probably goes on for a very, very, very long time.”

“If you're just doing the scaling project, you have to be one of those, or you have to be an effective subsidiary of one of those companies.”

“Pretty much all of the major gains that we've seen in the open source space have come from data improvements.”

“The current LLM API market is dominated by synthetic data.”

“If you're only selling models, it's going to be difficult because it's going to be like a zero margin business because there's so much price dumping.”

“Value is accruing beneath, like at the chip layer, because everyone is spending insane amounts of money on chips to build these models in the first place, and then above, at the application layer, where you see stuff like let's say ChatGPT, which is charged on…”

“Right now, chips are just exceptionally high margin, and there's very, very little choice in the market. That's changing. I think it's gonna change faster than other people think. But yeah, I, I'm very confident.”

“You can definitely train big models on TPUs. Those are actually now a usable platform for super large scale model training, and I think Google has proven that quite convincingly.”

“Because these models are getting smarter, humanity's ability to distinguish between them, or not humanity, but each individual's ability to distinguish between them becomes way harder. You can't tell the difference between generations, because you're not enoug…”