The Ledger, every show

“That it's very like that in our lifetime, neural networks will become capable of learning anything and everything that we are capable of as humans.”

“Well, the first thing that I think is at this point, almost uncontested, but frankly, when I was building source, most people didn't think would happen is the fact that the majority, if not all developers in the next couple of years in the world will adopt som…”

“We don't think we're more than three years out to that point where a non-engineer, someone with no coding, you know, software development background is able to come and give one of those high level vague things of what they want built, right? Cause that's what…”

“And it's that RL loop that is very interesting because since it's programmatic, since we have an Oracle of truth, we can scale this up far larger, right? Magnitudes larger than what you can do with human feedback today.”

“We didn't take Megatron or DeepSpeed or kind of the big open source frameworks for training large language models. As our base, we built our own from scratch and we built our own distributed training for our pre-training stage.”

“We've decided to scale up rev nets, reversible layers. So, so to the best of our knowledge, I think we're training the world's largest models using reversible layers that, that we've seen at least that nothing else that we're aware of publicly.”

“We went through about 3200 or 3200 people that we ended up putting on the list. And we had this location column, like think genuinely Google spreadsheet, you know, like where's someone based. And at some point we looked at all these locations and we're like, w…”

“I think it all just came down to one thing and I'll stop the monologue is the fact that I rather live in a world that has a hundred foundation model companies than a world that has five. Even if I was one of the five. And the smallest and most meaningful contr…”

“Model building is ultimately 90% engineering. And I think we all know it in the industry, because if you look at words, every researcher spending their time, they're spending their time writing code, right? Looking at data and writing code.”

“Laguna access two that we launched. It was five weeks from the beginning of pre-training to launch. The model that we're going to talk about today was eight weeks from start to pre-training to launch.”

“I actually think you can sum down. So I saw 90. Five percent of model building to just doing, you're just doing two things. You're improving data or you're improving compute efficiency.”

“A 118,000,000,008 B active model, which is not that large. It fits on a DGX spark and still runs at, you know, 3040 tokens a second on a spark is able to solve. Erdos three 97 independently. It's able to do complex programming tasks.”

“Mid-training exists because there's a mid-training team now, right? There's people or like people decide to focus on like a mid-training effort. But what you really want is engineering. And skill of experiments that allows for a much more continuous spectrum t…”

“You can't fine tune your way to success, right? Major capabilities emerge from training a base model made accurate and useful during fine tuning.”

“We think focusing on coding and long horizon software tasks is a path towards AGI because it forces us to solve the hard problems.”

“I don't think audio. Adds to that. I don't think it pushes us close to AGI. I think it is a necessary modality as you get close to AGI.”

“The difference of a model you could train on hoppers versus GB 300 is the difference in a trillion parameter model and a five or six trillion parameter model.”

“To get to highly capable intelligence that can reason, that can do planning, that can understand the world. You're almost better off putting a set of blinders and focusing on, on one proxy for that.”

“I don't think it's a winner takes all market, but it does seem to be that it's a small number of companies who are able to get there.”

“So Project Horizon, in a nutshell, is today announced a two gigawatt campus. We can actually go far beyond two gigawatts on the site that we're in.”

“And so what you're seeing from us is that before the end of this year, we're starting construction on a 250 megawatt data center. It's natural gas powered with grid connection as a backup, and it will house an incredible amount of compute, but it will be the f…”

“To energize, 250 megawatts is about half a billion dollars worth of physical infrastructure. So in the case of gas turbines, it's gas turbines, but there's different paths to it. Building a data center, like the powered shell that brings together all of the eq…”

“And what I found to be one of the best advice I got a very long time ago, whenever you find yourself in that situation, well, this is when you want to start finding the experts to join you, but how do you find the experts? Because you yourself aren't. And here…”

“When we're building 250 megawatts, We're doing it with a less than 450 person on site workforce with very skilled tradesmen and construction workers that we're bringing over there and we're housing over there versus a world where you might need traditionally 2…”

“So at the beginning of Q four, end of Q three, so next year, we've got the first compute coming online. And then we're finalizing in Q one, 20, 27, the first you know, 250 megawatts, but already next summer, the next 250 megawatts starts construction. It's a s…”

“Renewables only, in combination with the battery capacity that you would need to power a data center if you want renewable only, it's just not economically viable yet.”

“Models and agents to me are effectively the same thing.”

“It is not that I can linearly add more GPUs and train increasingly a more larger model. If I do so, the time it takes becomes exponentially longer.”

“But if you think about the Malibu agent as a coding agent, for instance, right now, it sits at a level of like sweet bench verified, for instance, where Gemini two and a half pro was when it came out.”

“Usable code for training is what we refer to as about three trillion tokens. And if you look at kind of usable language in English on the internet for training, we're talking about anywhere between 10 and 15 trillion tokens. There's a massive amount of code th…”

“We put it in an environment, say it's an environment with a 130,000 real world code bases, several orders of magnitude, the largest environment in the world.”

“The Blackwell generation for us from a training perspective doesn't unlock anything new. It just means that we have to, we can do more with a certain set of chips. My H 200 become less valuable in the world, but it does not necessarily mean I have to go upgrad…”

“If look at our capital raise, our last five hundred million dollar round, you'll see that there's none of the big hyperscalers, Google, Microsoft, you know, Amazon were part of the round.”

“There is one corporate that became part of our round, and that was very deliberate, was NVIDIA. Not in an outsized stake manner at all. And it's because we collaborate really closely with them On, you know, the next generation of their chips, on the software a…”

“They built a 100,000 GPU, three, 32 K interconnected clusters in Tennessee in the span of months, and XAI showed up with Elon's strength, the ability to build physical infrastructure incredibly fast in the world.”

“I have found no shortage of people in Europe that, or in America, there's a stereotype about Europe But the fact of the matter is that people who want to join races and do truly their life's work, they're built differently, and you can find them all over the w…”

“That it's going to, that progress is going to halt.”

“The reality of regulation in many cases is that it becomes an expensive bureaucratic overhead. And that harms the most young startups. It doesn't harm companies that have raised massive amounts of capital.”

“The world is finding a balance and the balance that I'd like to see that the world is, and I think it's moving towards it, is to regulate the end user application of AI.”

“Source code is one of the very few things that we generate with neural networks. That we can actually execute an introspect. It doesn't require human feedback to evaluate it. It doesn't require humans to step by step reason through it. We have built compilers,…”

“The capabilities of these products are determined today by the capabilities of the models underneath, right? About 80, 90%.”

“But in terms of what we're delivering to the end user, we're starting with a direct competitor to get up Copilot X. And so natural language, multi-turn dialogue system. Where developers give instructions of what they want done while they're in their editor. Th…”

“And what we missed throughout that entire journey that we were on the right track, but we should have just kept scaling up. And today to all of us, the scaling laws and scaling up seems like the most obvious thing, but having spent four or five years of my lif…”

“We started as an American company. We have always been an American company. And early on, we made a very conscious decision. We said, we're not going to hire any researchers in the Bay Area. We're going to actually look for talent everywhere else in the world.”

“When we first wrote our first training code base completely from scratch, it wasn't a fork of any open source. It was just like, okay, let's build it from scratch.”

“We're less than 70 researchers and other 35 engineers. And we're running, well, I haven't checked the latest count, but far more than 10,000, maybe 10 to 20,000 experiments a month.”

“One of my favorite metrics about like Laguna S is that there was no on-call events, right? Like completely zero. And actually we haven't had a meaningful on-call event, like something to wake up for as far as I recall this entire year.”

“Once we realized that we treated data as immutable and code as always version, and you could always track and trace every experiment end to end perfectly, you could repeat everything perfectly, right? You have perfect reproducibility. I can still reproduce run…”