LLMs

“This is an opportunity where college students are particularly well, like young founders are particularly well positioned to work in it because nobody has like there's no one walking around with like four years of LLM experience. So like everyone is starting f…”

“I definitely think that whole world of controlling, within an enterprise in particular, like, controlling who has access to, like, which LLM has access to, like, what data and who has permissions is, like, a really ripe space for building interesting software.”

“The stuff that I've seen in the batches are actually taking off is a little bit more mundane. Like it's I mean, I probably say a lot of it's sort of like workflow automation, like It's finding things where there was, like, a human doing some repetitive tasks u…”

“And yet we actually don't get that many applications for people working on this.”

“The bad ideas, the target ideas of yesterday could actually be possible to make successful now thanks to LLMs.”

“It is definitely the case that there is a data wall and that if you take the same techniques that we were using to scale LLMs you know, at some point you're going to run into that.”

“And now that that has been cracked at this point, I think we actually have a very clear path to just focus on scaling. You know, we were, you know, talking about that you know, the zero to one part, that's not about scaling. I think there's a really strong cas…”

“The LLMs are maybe going to push people to choose, because the actual writing of the code may become less important, and it's about, are you really, do you have taste and you want to solve product problems, or Are you an architect and you want to solve systems…”

“It's, I think maybe the reason why these models are not great design yet, it's like on the art side you've got diffusion, on the problem solving side you've got LMs, and it's not clear that people have figured out how to, you know, marry techniques together ye…”

“We found that it makes it a lot easier for LLMs to follow, because a lot of elements were post-trained in RLHF with kind of XML type of input, and it turns out to produce better results.”

“It's actually hard for today for even, like, the best LLMs to find those”

“I think this pattern of sometimes when it's too hard to even kind of write a prose around it, let's just give you an example that turns out to work really well, because it helps LLMs to Reason around complicated tasks and steer it better because you can't quit…”

“Well, one of the things we've been using LLMs for internally is actually helping founders figure out who they should take money from.”

“The model really wants to actually help you so much that if you just tell it, give me back output in this particular format, even if it doesn't quite have the information it needs, it'll actually just tell you what it thinks you want to hear. And it's literall…”

“They display jagged intelligence, so they're going to be superhuman in some problem-solving domains, and then they're going to make mistakes that basically no human will make, like, you know, they will insist that 9.11 is greater than 9.9, or that there are tw…”

“Mac minis, for example, are a very good fit for some of the LLMs, because it's all, if you're doing batch one inference, this is all super memory bound, so this actually works.”

“LLMs don't natively do this, and this is not something that has really been solved in the R&D of LLMs, I think. And so context windows are really kind of like working memory, and you have to sort of program the working memory quite directly because they don't …”

“LLMs, like, flip, they flip the direction of technology diffusion, ah, that is usually, ah, present in technology. So, for example, with electricity, cryptography, computing, flight, internet, GPS, lots of new transformative technologies that have not been aro…”

“I think we will start to see this on more general systems, be them LLMs or others, That we will find more and more scientific knowledge within them, and we'll use them for important, important purposes, and I think this is really where this is going, and I thi…”

“Protein structure prediction is in some ways far ahead of LLMs or the general machine learning space, and having blind assessment.”

“I think the LLMs by themselves are not good enough for that yet.”

“Whenever someone asks how to set up anything that you need, some sort of backend, Firebase type of transaction, the default answer from all the LLMs is actually a Superbase.”

“He has this concept of, like, the LLMs reaching the dumb zone, where it's like, after a certain amount of tokens It just starts, like, degrading in quality. And I actually think that's very true, especially if you think about, like, how the reinforcement learn…”

“At Anthropic, the way that we thought about it is we don't build for the model of today. We build for the model six months from now.”

“Whenever I see the use of emojis, even though they're not the system emojis, I feel like it's a little lazy. And so I feel like LLMs kind of take the easy path because they don't have any IP really themselves. They're just like big models kind of cobbling thin…”

“When there was a good website kind of establishing a trend, it took a while in the old world for others to kind of like copy these trends. But now with LLMs, if there's a good website with a purple gradient, it makes it into the LLM because the LLM gets traine…”

“There already exists some type of recursion that people are used to in LLMs, which is a chain of thought we mentioned earlier, but that is a recursion that's happening in the token space of the model's outputs, not inherent to the model itself. That's sort of …”

“What you actually paid for that you have to give up is this latent reasoning thing and this compression in the time direction. There is no compression in LMs. Every single decode that I do, I still have to retain the entire, you know, Shakespeare novel just to…”

“And so it's completely true that at test time, they are turn complete. And you can simulate all turn computable functions at test time.”

“And I think the biggest pitfall of LMs is you have no sense of How much training data the model has seen for the exact thing that you're asking it.”

“And a new way to do it is actually have the LLMs just write the code you need right now. And then you write 10 or 20 times less code, but it's way more customizable. It does, you know, more and is more awesome. Between token maxing and even like, you know, don…”