LLMs

“Laura cannot teach new language. It cannot it's sometimes may struggle to teach new techniques or new, new concepts. It does well into adding and refining existing contact knowledge.”

“No, but these models will get better and you'll probably fine tune a model to be a better judge. I think that's probably what's going to happen. So I'm like reasonably bullish on this because I don't think there's really a good alternative beyond you just huma…”

“That's one of those things where I think long-term, you definitely can. I think some people say you can't. I disagree. I think you definitely can. Just right now, I haven't gotten into work yet.”

“And so we ran a bunch of experiments, and all of them supported the hypothesis that it was memorizing the data set in a single thing at once.”

“I think there's a couple companies who make, Tens of millions of dollars of revenue from, yeah, from LLMs or diffusion models for porn”

“And I think you're going to see a lot less of people having to slave away learning these things. They just have to know the broad capabilities and then the LLM will take care of the rest.”

“The LLM space is, like, the opposite. Like, the switching cost of LLMs is zero, right? Like, if all you're doing is, like, straight up, like, at least, like, open source, right? Like, if all you're doing is, like, you know, using some, you know, inference endp…”

“I actually think like it's really clear today. Multimodal models are the default foundation model, right? It's just going to supplant LLMs. Like why did you just train a giant multimodal model?”

“Like de novo RL is like a pretty terrible way to get there quickly. Why are we rediscovering all the knowledge about the world? Like years ago, I had a debate with a Berkeley professor as to like what will it actually take to build HCI? And his view is basical…”

“The assistant isn't the weights. The assistant, the entity you're talking to is something drummed up by the weights.”

“And if we map this to what the LLMs are doing, They're probably not able to have genuine perception, because they're not coupled to an environment in which things are happening now. Instead, it's all asynchronous in a way.”

“At the moment, the idea that we build LLMs that are being coerced with good behavior is not really sustainable. Because if they cannot prove that the behavior is actually good I think we are doomed.”

“Today we have language models that are powerful enough and big enough to have really, really good models of the world. They know a ball that's bouncy will bounce, will, when you throw it in the air, it'll land, when it's on water, it'll float, like, these basi…”

“And I think the work around over, you know, overfitting on the test, I think is like that. 100% is happening.”

“Traditional machine learning has a real material role to play in producing a system that hangs together, and there are, you know, guaranteeable Like statistical promises that classical machine learning systems to include traditional deep learning can make abou…”

“And so, to be compute efficient at inference time, it's much better to train it much longer training time, even if it's an effort, an additional effort, than to have a bigger model. That's what I call, like, I refer to the chinchilla trap, Not that Chinchilla …”

“I think the biggest change in this market is like, Latency is actually not that important anymore. Like we lived in the past 10 years in a world where like 10, 15, 20 milliseconds made a big difference. I think today people will be happy to trade 50 millisecon…”

“Well, there are going to be some things that you need to build this when the tech does catch up. So retrieval being one of the most important things, like the model is going to have to be able to like pull code out for code base somehow. So we were like, well,…”

“If I have to spend so much time thinking about how I want to frame the question that it would have been faster for me just to get the answer. Didn't save me any time. And so oftentimes, you know, what I do is like, I just dump in whatever current thought that …”

“It would have taken me. You know, several hours to figure out some things that take 10 minutes if you could just ask exactly the question you want the answer to.”

“So I think way more than all of us, they are e-cons.”

“There are a number of papers which have found that the label of the exemplar doesn't really matter, and the model reads the exemplars and cares more about structure than label.”

“If LLINs are about to become much better at coding over time, then I think you can expect that the LLIN could actually do this for any custom application over time. And so the LLINs could act as a kind of compiler What you're interested in, they're gonna do al…”

“Sometimes I say that like the LLMs aren't [4137] Great at planning yet. [4138] So we can help them plan by telling them how to plan and code. [4140] Cause that's very explicit and that's a good way of communicating how they should plan and stuff like that.”

“If you build your system in a way that Kind of assumes LLMs will get better at reasoning and get better at sort of agentic tasks in the LLM itself. Then I think you will build a more durable system.”

“If you choose, like, the right algorithm and the right approach, you can actually get, like, super good results without having, like, a ton of data, and even with LLMs. You can apply all these other techniques to give them, to kind of bootstrap, kind of like t…”

“A lot of the synthetic datagen papers, like orca-three, wizard-lm, they show that models are better at verifying output than generating output.”

“Eventually, I think that that's where the world is going to. Like the code It's going to be there, and we're, there will be developers, et cetera, but as agent improves and capabilities of the LLMs and integrations to different parts of the environment, develo…”

“LLMs are showing enormous progress in reasoning, especially with the latest set of models that we've seen, but we're not really seeing, I have a feeling that VLMs are lagging behind, as we can see with these tasks that should be very simple for a human to do t…”

“When some very useful tool goes down in cost by 200 X in like the space of, I don't know, a couple years, There are going to be new opportunities in search, right? So like, to not integrate this and build up, to not like rethink search from scratch, the search…”

“We could get by, which we are right now doing, using, like, LLMs as the labelers.”

“And actually I found that that is a better way of doing this than when doing, you know, XML bracket, good example, close bracket, bad example, close bracket. Those examples tend to meet. There's an issue of prompts leaking, example leaking. Where there's a few…”

“The models have proven to just be far worse at reasoning than people sort of thought, and I think whenever I hear people talk about LLMs as reasoning engines, I sort of cringe a bit. I don't think that's what they are. I think of them more as like a simulator.”

“So I think AI gateways are essentially the operational platforms that enable teams to connect to LLMs more efficiently. They help you improve cost, performance, and accuracy by not having you to build individual connections to all of these different AI service…”

“The problem with knowledge graphs that we found is like, and I don't know if you can tell me what your experience has been, but they're great for representing the data, but then like using it at inference time is kind of challenging, like... Just like the LLM …”

“Basically what we came up with, it's essentially you just write a TypeScript function and you document your TypeScript function with JSDoc. And then we extract all the information from there and basically make that and pass that information to the LLMs.”

“I think techniques have stabilized. I think the kinds of failure modes of LLMs, I mean, they're still there, but it's not like changing every single day. We know that LLMs are bad at certain things. We know a little bit more about say limitations of the transf…”

“Another thing is that we actually combine it with LLMs. So the transcripts, LLMs and the speaker diarization, like bringing all of these together to recalibrate some of the switching points.”

“An agent is really like LLMs and a bunch of workflows and coordination and orchestration and then extra, like some sort of services, right? Like you have kind of the brain of the operation, which is the LLM and it can come up with a plan. And then, but then it…”

“We can get a dramatic multiple words of magnitude reduction by going to a lower model with literally like no change in the quality of the output.”

“Eventually, like, we want the LLMs to Deploy these services, apps that they are building, and, ah, have them manage it, and developer is more like in the backseat, like, looking at things if everything is working correctly. If your swarm of agents is working c…”

“The important part is that you don't need to explain the model, and the model doesn't need to care about how to keep the state of the program running. So it was, especially with our earlier models, I think the models are now smarter, but they kept producing, l…”

“So there's a, there was a lot of like integration work to bridge that gap, but now the LLMs, because they've taken on this tool calling and are getting better at tool calling, the story is the integration story has definitely gotten a lot easier. I think it's …”

“We are working on fine tuning a model. We think we can get the costs of inference way down and the quality and specific, you know, AI working in a spreadsheet is not really what these models were trained to do, right? There's a lot of two D positioning code er…”

“I think it's pretty clear that as these models get bigger, they have a world model, and that world model becomes better with scale. So they are implicitly developing a world model, and I don't think it's something that you need to explicitly model.”

“My hot take was, I don't know if we need things like React Native in the future, because you can just ask these LLMs to translate between different native environments.”

“Our last leaderboard is saying that models are really great at tool calling. So it's like safe, but they actually not, right? They're making mistakes and this is going to recalibrate the expectation of the users that Be careful because they're still not perfec…”

“And the problem with that is that we don't want to incentivize AI to derive the program that made the game. Right? And so if we continue to have humans make the game, then the AI is incentivized to try to reverse engineer the G inside of humans, and that's kin…”

“You can fight the battle, I think, of saying, OpenAI shouldn't consume your content without paying for it, or shouldn't consume it at all. But I think it's gonna be really tough to fight the battle of saying, like, LLMs writ large shouldn't consume my content,…”

“In the long term, we believe that LLMs are going to improve, and they already have improved to the point where you don't need to explicitly specify an at mention. The LLMs should be intelligent enough to pick it up.”

“Like when you're talking to these LLMs, the work you're doing is not writing code, it's reviewing code.”

“If you go back a year from today in July of last year, the ratios of like input to output for LLMs were very, very heavily on, on inputs where you could be doing, you know, 10, 10, 15 to one ratio of input to output. But that has completely flipped and it's ob…”

“The performance of LLMs is not invariant to how many tokens you use. As you use more and more tokens, the model can pay attention to less, and then also can reason sort of less effectively.”

“I think that, like, this is going to be the dominant paradigm. I actually think that, like, probably purpose-built re-rankers will go away, and the same way that, like, purpose-built, they'll still exist, right? Like, if you're at extreme scale, extreme cost, …”

“A lot of the features what we see, you know, where people build like elaborate workflows, like I have my custom slash commands and they trigger custom Custom sub-agents and they in turn trigger custom MCP tool calls behind which again another model is doing in…”

“Yeah, so prompt enhancer, that's a bullshit feature that doesn't actually work. The theory behind it is nuts, because what helps LLMs is not tricks and phrasing your prompt in a certain way. It's fundamentally information that you have in your head that you ca…”

“Because most of the data you're looking at is like time series data. And these LLMs are really bad at processing time series data, right? And that's really where like good statistics comes in.”

“The agent is not good at looking at time series data, so that's, that is what statistics needs to take care of. But statistics doesn't understand what is the relationship between latency and memory usage and disk utilization. So that is the LLM part.”

“It forces you to think deeply. It forces you to structure your reasoning. I don't know about you guys, but when I read a doc, when I write a doc, I've like read the doc like 50 times and thought about like, Does this logic track? Are there gotchas I'm not cons…”

“And then like you actually lose something if you translate to this like purely tokenized representations that we use in LLMs, right? Like you lose the font, you lose the line breaks, you lose sort of the two D arrangement on the page. And for a lot of cases, f…”

“I think that there will just be labs coming at this problem from both sides. And everyone ends up in roughly the same place, and the same place will be whatever people think is cool.”

“Honestly, Jan's podcast that he did, I don't remember which one it was, but a long time ago, where he basically proclaimed LLMs to be a dead end it was one of the things that inspired me to do this.”

“RL, the way it's applied to LLMs right now, is kind of a weird, funny tool where it doesn't really generalize beyond the training distribution that much. It generalizes to some extent, and generalizes in interesting ways, but It's like very peaky, right? Like …”

“I think where we're getting to is that these models have gotten smart enough, they've gotten better, better tools that they can perform better when just given a minimalist set of tools and let them run, let the model Control the agentic workflow rather than us…”

“The more you go into building something using LLMs, the more each bit of what you're doing ends up being a benchmarking problem.”

“My intuition has been that trying to craft LLMs into deterministic workflows and DAGs is, is kind of underselling like the power that they have to actually plan and execute more in a more sophisticated, like fluid way.”

“We can actually automate so much of the scientific method, because it turns out, especially in a field like biology, which is very empirical limited, you know, the top one percent guesser of, you know, what they think will happen in experiment that, you know, …”

“Part of the reason the LLMs are so large isn't just because of their reasoning capability, but it's also because of, like, the sheer quantity of information that they store. And I think here there's a little bit less of that, you know, and I think it's more ab…”

“My take is that there are four, four things where if you can cover all four of these things, LLMs are not like three X faster or five X faster. They're like a hundred X faster.”

“Taste, creativity, sense of individuality is still something that I think that the LLMs are not producing out of the box. And I think that's going to be an interesting frontier.”

“So the way I think about it is we've had four fundamental breakthroughs in functionality, LLMs, reasoning agents and then and then now RSI and they're all actually working.”

“If you ask me, would I be confident that in 20 years we'll have these sort of like static documents in which we publish our results as papers? I would think not. Like that doesn't seem like the best thing we could be doing. Maybe some kind of interactive paper…”

“I feel this is actually, this part of long contacts is a little bit ahead of the LLM part.”

“I find AI in like what most of this like launch here is, is actually like less building forward. It's actually like. A recursive loop backwards. I'm always looking at what had happened first, like go back through the week and tell me what we did, what worked, …”

“What we saw is the moment you send the result with the repair logic, right after that, the third tool call is fixed. Instead of, you know, it all of a sudden becomes super smart. It understands like, okay, I got the result, what I was looking for, and I'm gonn…”

“I personally don't use OKLCH, but apparently LLMs are really good at it. And if you see them using HSL or something, they are, they don't actually are able to control the lightness in HSL very quickly, but on to human eye, it's very, very easy to see like this…”

“Like if they are seeing a lot less tool call errors, they're much more creative. They are, they can explore a lot and they can continue a lot longer.”

“Or you can even train a model which is able to natively understand crystal structure representation, and like image modality is a modality for LLMs these days. Crystal structure can be a modality for LLMs as well.”

“The demand that we see for something like that is actually not for LLMs. Although sometimes people want to run evals and do synthetic data prep and there it makes sense. But it's from a lot of non LLM companies like people who are doing computational bio, like…”

“Yeah, probably LLMs are going to run out at some point and they're not AGI and okay, we have maybe another 30 years of AI winter or something and then like the next paradigm really is actually the thing.”

“And, you know, I think this class of models is going to be like just as big, just as impactful as LLMs, but it's almost like the compute market, like kind of doesn't realize that yet, both in the capacity sense, but also in like the software stack sense.”