language models

“All of us have nightmares about writing beautiful soup code in order to parse web pages. It's basically regular expression parsing over ever-changing websites. It's horrible. We have done a bunch of it in the first two-ish years of Neva. That stuff is also eas…”

“If you think about predicting the next word, It's, it seems very simple, but you have to really internalize a lot of what is going on in this context. What are the previous words? What's the syntax? What's who's saying them? And all of that information and con…”

“You know, people say that sometimes all language models just memorize because they're so big and train on clearly a lot of texts, but these examples, I think really indicate that there's no way That these language models are just memorizing because this text j…”

“If these models start interacting with the world and accepting external inputs, now you can not only just sort of jailbreak your own model, but you can jailbreak other people's model and get them to do various things. And then, so that could lead to sort of a …”

“So I think there's pre-training, which is predicting the next word and developing a world model, so to speak. And with those capabilities, then you can, you still have to say don't hallucinate, but it will be much easier to control that model if it has a notio…”

“I feel like we can do these things even better now with language models. So yeah, I think it'll happen in a year, more than five years.”

“The two big problems with it are number one, like the knowledge is not up to date. You know, it's only, it only knows stuff it was strained on. And number two, a lot of the things it says are inaccurate and it's confident, but like wrong in various ways. And I…”

“There are some papers that I can point to later in the show notes that kind of show how memorization scales with the number of times something shows up in the corpus, and you have to hit a certain frequency before these models can kind of accurately remember t…”

“I think that might be harder in video and I wonder if there is something that we are kind of fundamentally missing in terms of how we approach video generation. So it's not quite answering what you asked me, but I do think that it might be a little bit slower …”

“I think we'll see, you know, first organizations like this, probably even this year where Potentially with a simpler mission, some kind of more straightforward like KPI metrics, but where kind of this information propagation and onboarding of people happens al…”

“The fact that this model is, like, doing a really effective search in kind of this knowledge space means that probably, like, pushing more into that concept is more useful than doing more searches at inference time because, like, it means you already lost all …”

“Like, we know even in theoretical senses, like, they cannot learn to do multiplication in the general sense because it literally doesn't fit in the context window, right? Like, multiplication, they can learn to do addition in a modular sense, and they can lear…”

“Fundamentally, SaaS, software is all, we're all in the same information, people, paper pushing activity, right? It's like a piece of paper coming in front of you. A human, like, changed a couple bits, pushed to another human. Language model can do some form of…”

“Now the new technology happening is AI language model, and if you build more with it or just think more with it, language model wants information to be one place. Once the endpoints to be connected, so it's easier to, is it hard enough to like a conversion lan…”

“I think there's bundling of distribution and bundling of information. What you're describing to me is more of Microsoft, more like bundling of distribution. Language model wants the bundling of data, bundling information.”

“I probably feel it's probably less the agent side essentially bet on language model, that's the communication.”

“I think on the foundation model side, from a venture capital or angel investor perspective I think we're going to see fewer new language models But we should see models in a lot of other areas and, you know, we have new things happening in music. We talked a l…”

“Transformers are definitely, I think transformers are definitely like the right direction, but I don't think that we're going to get a lot of you enough utility if we're not like somewhat trying to figure out a way to combine the great, amazing knowledge of li…”

“I am more confident actually that or I guess I would bet that the solution arc probably comes from an outsider. I think it's probably gonna come from somebody who's sort of not indoctrinated in the current way of thinking about language models and scale.”

“Just sort of objectively, language models do not work to beat Arc. And people have tried.”

“Scaling language models purely will not get there.”

“So then you, of course, are absorbing all the knowledge on the internet and then sharpening those models around being, following instructions, being aligned with humans, but you still have this Probability distribution that will assign non-zero probability to …”

“If checking that something is correct is easier than creating the solution, then we're in business because the language models will be able to evaluate their own samples more accurately than to generate them. And then we have a sort of reinforcement learning l…”

“To accelerate that, that sort of progress, you want to start sort of really exploring what's the reasoning the model has, and by making it more redundant, more logical, by iterating more on these kind of ideas, you could imagine generating a very small program…”

“Well, I think all language models are quite sensitive to prompts, to the way that you present data. They all have their own individual quirks. The way that you talk to one might not work for the way that you talk to another. And so when you're building a syste…”

“I feel like actually people underestimate how much intelligence goes into motor control. Many, many years of evolution is what led to us being able to use our hands the way that we do. And there are many animals that they can't do it even though they had so mu…”

“Anytime you go into the physical world, it's always harder to generate data, and that's one of the reasons that the language models where you just effectively collect the wisdom of the internet digitally are the first places where we've really seen this scale …”

“Like, even back in my PhD, I think one of my, I guess, less well-known works was actually about, we showed that you can train language models to simulate different kinds of students. For tests. Yeah, yeah. And by simulating students, you can actually design be…”

“We've actually done the math on this. And so for a simple application on our platform, it would be 10 times greater in cost to try to replicate it with a language model.”