The Ledger, every show

“We're like relatively small. We're a 10 K H 200 cluster company right now. We'll be scaling to a lot more soon, but and really a lot more.”

“Laguna S Laguna Small, 118, one, eight billion total parameters, eight B active. So very sparse. It's a scale up of the XS architecture. It's the kind of classic or quite classic these days, like three to one ratio of sliding window attention to global attenti…”

“No doubt it's going to be better in your own harness. And it's just because of like, where are you putting your reinforcement learning compute, right? You're putting your RL and your synthetic data. You're putting it to your own harness because it's the one th…”

“We're, I don't think we'll touch audio for a very long time.”

“I think NVFP four is, you know, underrated in terms of what it is.”

“It's not very helpful to have a post training recipe for a smaller model and try to apply it to a bigger model. Yeah. It just, in all cases, you're gonna have to rethink most of the recipe. But recipe for post training for a bigger model applied to a smaller m…”

“There's three layers of the stack that fundamentally are going to matter. It's energy, it's compute, and it's the intelligence built on top.”

“They are the anchor tenant on our data center.”

“What was particularly interesting to learn about the infrastructure space over time is that it's an incredibly small world. Everybody knows each other. It makes tech feel like a big world, right? The construction of data centers at scale has really only been d…”

“And I think what we'll see in the next couple of years is a lot of model companies ourselves as well. We go into these enterprises. We find high value use cases. We use our agents to get to 60 or 70% You know, of quality, and then use additional reinforcement …”

“Foundation models today can't do it, and we don't yet have a path In them successfully doing so in a way that, like, really improves it.”

“And so we're starting to open up much more broader outside of software development and come back to what we've always said, like we want to be able to power all of knowledge work in the world.”

“That's our upcoming family of models, Laguna, where we'll have a small, a medium, and a large. So small is finishing training in a couple of weeks, medium actually starts training this week and large starts training as our 41,000 GB 300 come online.”

“So we today operate on workstations that go into SCIFs. We operate on servers that are, you know, on-prem or even as far as air-gapped.”

“And still today, if we look at where, like, researchers for pool sites sit, they sit predominantly, you know, 95% outside of Silicon Valley.”

“Models reach the frontier. That's right now. I think we see a straight line towards this.”

“We're going to look back on this moment 10 years from now, just like we would look back to the moment of mobile, internet, and realize that that was the moment where the table got set.”

“And we've got probably decades, if not, you know, hundreds of years of improvements still left there and different forms of it over time.”

“If you have something that can determine an oracle of truth that can help say, this is better and this is worse, or this is correct and this is wrong, that's when you can actually use synthetic data.”

“I often refer to it as a drunken bar fight that's happening in our industry, is that we are, you know, all these companies are massively incentivized to drop the cost of their models as quickly as possible.”

“Models today, in my opinion, are pretty much there. Maybe there's a tiny bit left to say, but we've closed that gap, you know, into an incredible amount.”

“The 10,000 GPUs that we've now brought online this summer, you know, that came from this capital, allow us to make incredible advancements in model capabilities”

“The creation of models is CapEx. The operating of them, the inference to running of them is OPEX. But the OPEX to run them requires extremely large scale physical footprint in the world.”

“I think this is the, one of the largest build outs that we've seen in physical infrastructure since, you know, the last couple of decades in, in the cloud.”

“The data centers from, you know, two years ago versus the data centers in terms of size and power requirement that we're going to see in the next two years look radically different, not just because the scale of number of servers and nodes that we're interconn…”

“For inference, we don't need all of the machines to be connected to each other in the same place. For training, we need them all to be connected to each other in the same room, in the same place.”

“The reason I mentioned those three specifically is that they are all building extremely large volume of chips. And constantly iterating on faster and better and better generations of chips for training and for inference. They're, for me, the three primary play…”

“Pretty much what we've seen consistently with every two-year generation of training from NVIDIA is about a two X, one and a half to two X, you know, performance increase. But training is about two X every two years.”

“The nature of large technology companies choosing to invest, you know, in frontier AI companies is, is frankly the game theory optimal thing for them to do.”

“I think Elon is one of the most impressive examples of someone who has done this For such a prolonged amount of time, at moments in time when the entire world, you know, refused to align around his view, I think there's companies in the world that get built be…”

“AI for software developers. I don't think anyone in the world anymore questions that software development moving forward is going to be a, for the foreseeable future, a developer-led, AI-assisted world, an increasingly AI-assisted world.”

“While there's still a bit of a gap, I think image generation is one already where we're seeing more and more commoditization over time.”

“The number one company we have to give credit to is DeepMind. DeepMind built an incredible talent base, and they built it out of London.”

“But I still actually look at this thinking that there will be more value built on top of us in the future than what we can possibly unlock only ourselves.”

“It was probably the dumbest financial decision of my life, considering it was an all stock offer and GitHub sold to Microsoft, I think, less than a year later.”

“I think very few people from the investment landscape have taken a strong conviction on what technology is going to bring in the next 10 years, and then found a way to place bets all over the globe.”

“And we had a lot of the things running things that today that look like copilot where you could, you know, get code suggestions for based on natural language instructions.”

“Software development in our opinion, being a pretty good proxy task. For a big part of the spectrum of intelligence. You need an understanding, a model of the world, because we built software for the world. You need to be particularly strong at reasoning and y…”

“They're the first people to leave you if something else is more capable.”

“Our data split is about 50, 50 between language and code.”

“We failed ultimately at the time, and it kind of was like biggest failure of my career, right? You blew twelve million dollars of investors' money, which was a lot back then.”

“They started years before ChatGPT.”

“One of the guys on our team who started as an engineer building our agents is a legit reinforcement learning researcher now making real progress. And that happened in the span of like six months that would have not been what I think most people assumed was pos…”

“Laguna S was trained in FP-A, Only thing that in this run, I have to admit, that wasn't FPA. It was the all to all. In the new run we just started yesterday, the FPA was all to all.”

“The new medium started training, and it's much bigger than the last medium started training yesterday. So it's a 39 day pre-training run.”

“My biggest wall clock bottleneck right now is RL time. And it's just because I can scale it up further because I can't add more GPUs to it because of that size constraint.”

“When we talk about this big multi-gigawatt project, we've got about eight years worth of construction that we are planning out there”

“Right now we have over a million of those environments and tens of millions of revisions, putting us in like the thirty million range.”