The Ledger, every show

“Whereas, oddly, the US companies that used to lead the way have all drawn up the drawbridges, and that's gonna cause China to keep moving faster, because when you're in that more, both collaborative and competitive environment, you just Go way ahead, as we've …”

“They kept on kind of exponentially increasing the amount they were spending on their models, whilst the Return, you know, the kind of utility of those models was only increasing logarithmically, and you kind of very quickly hit this point where it's like, oh, …”

“It's a thing where you get most of the juice out of it in the first year or two, so we're still in that. Phase at the moment, and we'll start to hit the curve off point pretty soon. Just like we did for training.”

“I don't think we have any more evidence that ASI might be close now than we did 15 years ago, 15 years before that.”

“The amount of people and money being put into AI, it's going to keep going up, but again, the low hanging fruits being done now. So yeah, I think we'll continue to see the tailing off of growth.”

“It's very different to a research lab, which is like, hey, let's try and get lots of citations on a paper in a field that's so highly recognizable to its peers that they recognize it as being something they want to appear at their next conference, you know, wh…”

“Devon's an agent you know, a bunch of tools, tool calls, and tied together with an OpenAI model, if I understand correctly. Yeah, and it actually ended up definitely supporting our thesis, which is, there are so many places we wished we could have got involved…”

“I have no particular reason to believe that will happen, and for every point until we get there, if we ever get there, by definition, there's going to be humans interacting with AI, and so that's what I care about, is how do we do that In an optimal way.”

“I organized a chat for both of us with Kate Metz in the New York Times, and Kate Metz answered, sorry, and Alec answered this question for Kate, and Kate just like, so how did, you know, GPT come about? And he said, well, I was pretty sure that pre-training on…”

“So Code Llama was a, I think it was like a five hundred billion token fine-tuning of Llama II using code. And also prose about code that Meta did. And honestly, they kind of blew it. Because Code Llama is good at coding, but it's bad at everything else.”

“Then in the end, you know, Google didn't follow through, which is fair enough, like, asking everybody to, you know, learn a new programming language is going to be tough, but, like, it was very obvious, very, very obvious at that time that TensorFlow II was go…”

“But I mean, in the meantime, I will say, you know, Google now does have a backup plan. You know, they have JAX, which was never a strategy. It was just a bunch of people who also recognized TensorFlow two as shit, and they just decided to build something else.”

“RAG is like such a inefficient hack, really, isn't it? It's like, You know, segment up my data in some somewhat arbitrary way, embed it, ask questions about that, you know, hope that my embedding, you know, model embeds questions in the same embedding space as…”

“This group of, which has averaged about 10 to 12 people, currently nine, I think, have built a pretty Transformational and complex piece of software, which we can do a quick demo of later if you're interested. Using a complete web application development platf…”

“The autoregressive nature of language models means that if they make a mistake, and you correct it, and then say, no, that was a mistake, please do it this way instead. The more often you do that, the worse the dialogue answers get. Because it's in the trainin…”

“The nice thing is that with the dialogue engineering we discussed, the longer your dialogue is, the better the AI gets, which is the opposite to what we're used to, right? Because you can edit the outputs that aren't great.”

“For me, there was no technology DeepSeek moment.”

“I think they're shutting down that product or they've shut down that product, if I understand correctly.”

“Much to my surprise, the academic approach is, was way less successful.”

“If we're going to have 12 to 14 people create five to 10,000 extremely commercially successful products, we're going to have to be extremely efficient, you know, at every level.”

“If you're training for random weights, you better have a really good reason, you know, because it seems so unlikely to me that nobody has ever trained on data that has any similarity whatsoever to the general class of data you're working with, and that's the o…”

“So the point at which they're doing proper continued pre-training is the point at which that becomes a continuum rather than a phase. So the only difference with what I was describing last time is to say, like, oh, they should, you know, There's a function or …”

“This didn't make sense to have like a so-called non-profit where then there are people working at a commercial company that's owned by or controlled nominally by the non-profit where the people in the company are being given the equivalent of stock options. Li…”

“Companies are sociopathic, like, by design. And so the alignment problem, as it relates to companies, has not been solved. Like, companies become huge, they devour their founders, they devour their communities, and they do things where even the CEOs, you know,…”

“Like, 80% of the time, I find out the person has a really unusual background. So, like, often they'll have, like, either they, like, came from poverty and, like, didn't get an opportunity to go to good school, or they, like, you know, had dyslexia and, you kno…”

“We don't have any managers. We don't have any hierarchy from that point of view. So, for example, I'm not a manager, which means I don't get to tell people what to do or how to do it or when to do it.”

“People are building too many vanity foundation models rather than taking better advantage of fine-tuning”

“Now, the interesting thing is, you see, unlike Kaggle competitions, that decoder models still Are at least competitive with things like DiBerta VIII. But they have to be way bigger to be competitive with things like DiBerta VIII. And the only reason they are c…”

“There's a whole model that's been trained in a different way. So there's probably a whole lot of tasks it's probably better at than you know, GPT and Gemini and Claude.”

“To explain, it's not that you shouldn't merge models, it's that you shouldn't be distributing a merged model. You should distribute it a merged adapter. 99% of the time. And actually often, one of the best things happening in the model merging world is actuall…”

“We want to create thousands of Commercially successful products at Answer.ai. And we want to do that with like, 12 people.”

“Much to my, you know, horror, the story around creating web applications is much worse now than it was 10 or 15 years ago, in terms of, like, if I say to a data scientist, here's how to create and deploy a web application, You know, either you have to learn Ja…”

“It's like a convenience over the top of this incredibly complicated system that full-time, sophisticated software engineers have designed over the past few decades in a totally different environment as a way to build software, you know. And so we're trying to,…”

“I thought, okay, so if I do this at a much bigger scale, using all of Wikipedia, what would it need to be able to do to finish a sentence in Wikipedia effectively, to do it quite accurately, quite often? I thought, geez, it would actually have to know a lot ab…”

“And so we ran a bunch of experiments, and all of them supported the hypothesis that it was memorizing the data set in a single thing at once.”

“Even though I originally created the three-step approach that everybody now does, my view is it's actually wrong, and we shouldn't use it.”

“To me, the right way to do this is to fine, fine-tune language models, is to actually throw away the idea of fine-tuning. There's no such thing. There's only continued pre-training.”

“ULM fit is the wrong approach and that's why we're seeing a lot of these You know, so-called alignment tax, and this view of like, oh, a model can't both code and do other things. You know, I think it's actually because people are training them wrong.”

“There's a whole lot of technical debt everywhere, you know, nobody's really figured this stuff out because everybody's been so busy building what we know works as quickly as possible. So, yeah, I think there's a huge amount of opportunity to, you know, I think…”

“There is a thousand flash attentions out there for us to build. You just got to make it easy for us to build them. So like Triton definitely helps. But it's still, Not easy. You know, it still requires kind of really understanding the VPU architecture, writing…”

“Built our own fully integrated professional services firm providing legal accounting, bookkeeping. They're all using Solvert, by the way. They're all using our AI. And we were their first client, so all of our back office and everything is being done by a comp…”

“We've made it easier to add tools to the LLM than anything exists in the world. Literally any Python function that you put right there in that system is immediately a tool”

“We want to make sure at the end of building something, two things have happened. The first is you understand every line of code fully, why it's there, what it does, and that you've learned from the process. So at the end of this process, you're a better develo…”

“Quickly realized, actually, we've gotten so far off the, you know, academic normal research check now that No one cares about anything we care about on the whole and vice versa.”

“But I think it probably will work. And I think that'll probably be a significant jump in performance when you can sketch out the entirety of the solution first, and then fill in the blocks and gradually increasing levels of specificity.”

“Humans and computers work better together than separately, and then there are various things that kind of come out of that, and one key one, for example, is that the human and the computer should be able to see all of each other's work and be operating in the …”

“There are certainly situations where, like, Scripts, spreadsheets, VBA, access databases are useful, you know, but nobody like builds them as the components of their technology, you know, strategy. They're the things you use for little tools. It's small, well-…”

“Definitely I feel like we are more, I can tell we're more advanced users of AI than anybody else, because people can keep complaining about all the problems with AI, and I'm always like, I don't have any of those problems, you know.”