The Ledger, every show

“So the people who are pushing this narrative are really the people who are benefiting from this being the narrative. So these are the incumbent AI companies who are, who want to have either the market regulated. Right. Because then they benefit because they ca…”

“It's not even really about language models anymore. That has almost been solved, right? That's kind of why you see things plateauing off a little bit as well.”

“Long context models are inherently incredibly wasteful. You're paying for all this compute, and that's maybe why some of the companies that are trying to really sell long context model, long context window models, they will make more money from that, right?”

“I would like it to be true that with open source, we could just keep up with all of that. But I think that's just incredibly naive. Open AI has this very deep understanding of how people want to use language models. Basically nobody else has, and they have thi…”

“I think in terms of data modes and maybe you, you've seen this come by actually, there was this Google memo from an internal Google employee who had written that open AI and Google have no mode. I think for me as a AI researcher, when I read that memo, I was l…”

“Probably the chance of me getting hit by lightning, like right now is much higher than that happening.”

“What Europe is going to try to do is overregulate everything and just completely destroy innovation.”

“If you look at how OpenAI and Anthropic and these places define AGI, it's as systems achieving capabilities that allow them to effectively do the work of humans for the majority of economically valuable human tasks. Then we're not that far away from it. And so…”

“So if all the stars align, then OpenAI and Anthropic and all of these places, they had this great first generation technology. So they're kind of like the Lycos and Alta Vista of search engines. And the technology we have is more like PageRank. And that would …”

“We have kind of an existence proof now that it's actually not that hard to do this and so you don't need to invest all that much in, in data, and you can use synthetic data and get a pretty good model out of that”

“So I would guess that they spent at least a hundred X The amount of that, that single training run, right?”

“One common misconception about fine tuning is a lot of people think that you can inject new knowledge into a model using fine tuning. And that is not true.”

“You can build a very awesome demo on a couple of PDFs and things will probably work. But then you have to scale it up to a million PDFs, and then everything breaks down. And the reason for that is that a lot of these kind of advanced RAG systems still actually…”

“When are we getting to a hundred percent accuracy? And I had to give them the bad news that probably never.”

“Very often they think they can do it in house. And so I think that belief is probably going to go away in the next couple of years where people realize that this stuff is a little bit more difficult than they, Initially thought.”

“That money is temporary and it's going to dry up. And so it's, we're going to have a couple of cool pilots and demos, and then at some point it doesn't work and we will move on to the next hype train.”

“And so we think that's not what we now call a language model. So it's not one big parametric monster. It's something a bit more elegant that has the retrieval kind of built in. It's a retrieval augmented language model and is strained end to end so that it can…”

“You don't have to fine tune your model. It feels very intuitive. We have this great data set. We own it. It's our data. So we need to do something useful with it. So we need to fine tune our own language model. And so the companies who are offering that servic…”

“I think Sam Altman had this interesting quote where he was saying that he thought models would stop growing in size. GPT-IV kind of hit this ceiling. I think that's probably right, but not really because size doesn't matter. It's just that data size matters ev…”

“One of the use cases I've been seeing now for GPT-IV is actually that people are using it to generate data and then they're training on that data with cheaper models. So GPT-IV might end up disrupting, not knowledge workers necessarily, but it might just disru…”

“So PyTorch really without PyTorch, none of this stuff would be happening right now. And so it's really like fundamental for all of the AI breakthroughs.”

“And I think you're totally right that that's a much easier problem to solve much quicker and then slowly grow with the capabilities of these models.”

“It's just that data size matters even more than model size. And I think the Lama paper out of Meta really brilliantly showed this. Where if you train a smaller model on more data for longer, then you get a better model. So you get more bang for your buck if yo…”

“And so, so GPT-IV might end up disrupting, not like knowledge workers necessarily, but it might just disrupt like mechanical Turk and is just a, an annotator on steroids.”

“And they haven't even really trained as far as I know on the data that comes out of ChatGPT going viral, right? So they had ChatGPT, it went viral. This led to this giant, giant data mode that they haven't even really used, used yet.”

“Data contamination where a bunch of these language models are trained on the things that they are being evaluated on. So GPT-IV looks like it's an amazing coder, but it might also just be trained on the data that it's evaluated on, which means that it's not ac…”

“And so that's a, an interesting part of the market, but I don't think that, that the actual foundation model builders like open AI and contextual are going to build that technology in house. It's probably gonna be an external sort of audit.”

“This whole flourishing that you see right now of open source models that basically comes from Meta's generosity in giving Lama away for free. And if they hadn't done that, then you wouldn't see that.”

“At some point there's going to be a disillusionment with the technology and then funding might dry up and then these places are really in trouble.”

“The strongest belief I had that turned out to be wrong is that I really underestimated how important scale is in artificial intelligence. So I, and I think this is really one of the things that open AI has excelled at is that if you throw an order of magnitude…”

“I mean, you could argue that GPT-IV-O is also already a reasoning model. It just hasn't been trained on reasoning specifically, but, ah, it can do chain of thought, right? So if it can do chain of thought, it's basically already a reasoning model. It just hasn…”

“Where we're headed is that we will have more specialized language models.”

“So I would say, and maybe I'm biased because one of my best friends is, is on the original attention paper, but that was the real breakthrough. It's just like figuring out that you have this attention mechanism that actually allows you to yeah, to do a much be…”

“In the initial paper, we used a vector database or a face. So the words vector database didn't exist at the time. But so face was the first vector database.”

“Really focused on retrieval because that's really the only way you get these agents to work on your data and your problems.”

“We think language models are great first generation technology, but they're not quite ready. We see a lot of frustration in the market around that.”

“If you train a smaller model on more data for longer, then you get a better model. So you get more bang for your buck if you train it on more data rather than having more parameters.”

“We think it's still very early innings in the game, so I think a lot of people sometimes think that it's you know, the game has been played, but it's just getting started.”

“We're specifically basing it on this retrieval augmented generation, which is something that me and my colleagues at fair came up with in.”

“So we're not going to be able to really know why a neural net, what does what it does at the scale that neural networks operate at.”

“So I think in some cases it is a feature. If you want to use a language model for creative writing and if you want it to be really, really creative, then you probably want it to hallucinate. So in a way it's a spectrum of groundedness and hallucination where i…”

“At this particular point in time, probably yes, but just because the field is moving so incredibly quickly. And so I think that in the next year, we're going to see lots of other models coming out. And if you can have a language model, agnostic AI company that…”

“So if you look at anthropic and open AI, I think they're really chasing for this idea of AGI and they're relatively consumer facing.”

“I think there's a giant opportunity in the market actually for. A startup or several startups becoming like the Moody's or the SMP sort of you know the folks who evaluate the quality of AI for specific use cases because nobody really knows.”

“I think some of the rounds were pretty big, but I think it's also justified just because this stuff is really going to change the world. And so one bet, if it's right, has massive payoff.”

“So far I haven't really seen a lot of interesting things coming out of Apple.”

“And there's an auto GPT thing that is going to change the world, but doesn't actually work.”

“Why RAG became the way you name these things is because it's generative, right? So we were the first ones to have a generative model there.”