The Wisdom Wall

“one thing I've seen is that if you have, uh, An engineering identity that's based on that, like a value based ranking of difficulty or skill, like the, the specific types of engineering that are considered, you know, impressive or high status can actually be like less hard for agents.”

“if you can generate an enormous amount of code and no one is reading it, you don't know the quality of the code. Nobody deeply understands the code base and there's more fragility, right? It's like the slop problem, but instead of it being like vibe coding slop for random websites for non-technical people, it's vibe…”

“And, and if you are meta and you have Somewhere between, you know, 22,000 GPU clusters and 350,000 GPUs, um, available, uh, then continuing to train past, like, supposedly optimal points, like, does improve performance apparently, um, and doesn't just fully asymptote as soon as maybe a lot of the research community…”

“at the early stages, a lot of the data doesn't exist, right? Like you'd have to capture real world data. You have increasingly meetings over zoom, but, uh, you would want to capture a lot of information about people. Um, so much of it is access and the information about who is like leaving and like a hundred X engineer…”

“Because if you're on the wrong side of this secular change, and you can't get to the other side of it, you should in fact, like, sell. You don't have good ideas about how to be on the right side of history.”

“I actually think the idea that actual production of code becomes not the bottleneck for, um, if you know what the spec is, not the bottleneck is like incredibly interesting, but I, I, I do think it overstates like how much of the overall software vendor problem that is.”

“One like mistake that entrepreneurs and investors make that I have made is like you look at something and it's not working and like the issue is it is like a technical issue and then you like assume it's not going to work. But I think in AI you have to like keep looking again and again because stuff can begin to work…”

“I do not want a bunch of, could be conceivably very talented people, like between me and the founders that I, you know, owe support and partnership and work to, because I don't have any of the context. I don't care as much as I do. Um, and I think it's like, uh, You know, not a great experience when there's like four…”

“But in, in AI companies today, models are such a big piece of the product experience and that the. Uh, thus the like model design. And I do mean that in the aesthetic sense of the term is quite important and often comes from the founder. And I think an example, a bunch of examples of that could be, um, mid journey,…”

“today to train these large models, you, you need all of the GPUs co-located because there is enough, um, Uh, data transfer between different chips, right? Between your nodes. And there's a physical constraint on that in that you need to get that much power and to a data, data center.”

“I'd say generally nonprofit governance is not every organization, but as a class known to be kind of a Right. Because performance is hard to measure objectively. And so it often ends up more about politics and specific relationships and status games than outcomes.”

“alignment research is very tied to capability research, right? And so it's sort of impossible to be like, we're going to stop making any progress on research. But figure out how to control this stuff.”

“Um, and like, I, I think like being able to rethink market size is just still like a key underpriced investor skill right now at any stage, right?”

“if you have people who think about scalability and you give people with taste, um, and understanding of your business, like the ability to do direct manipulation, Right. I make these videos myself or whatever. It's probably going to be a lot more effective, but it breaks a lot of norms.”

“I have some friends running, let's say, the current generation of, you know, these people, too, of, like, sort of tech-eating services companies in traditionally very fragmented industries, and, like, in a small room, they would say, man, like, Sarah, Elad, the automation works, and the distribution is the problem,…”

“You go to many of the incumbent, um, software and services vendors, and despite having done this task, you know, for years, they have not actually captured the information you'd want to teach a model.”

“in, um, like, media platforms in general, they're, like, the ratio varies, right, but there are a lot more, uh, readers on X or consumers, like, people who scroll a feed on TikTok, or not TikTok anymore, I suppose, but whatever it is, than, um, than creators, and so I think, like, One thing that is just like unknown is…”

“in robotics, You know, most machine learning people will look at it as a, a data collection problem where, uh, your internet data, even video data of a bunch of actions just isn't, um, enough. We need, uh, embodied action data, like, you know, controls data in some way.”

“One thing you learned from the last generation of video applications, uh, was like, if you take, um, certain, uh, uh, dimensions of creativity, uh, away, or you, you just control them, like, and you, uh, lower the required quality of, uh, different dimensions, you make it much, much easier for people to create.”

“I, I think in, like, in the new era of trying to create experiences and fighting, like, all these, like, new application companies are fighting latency. As a, a primary consideration, because you have, you have the network, the models are slow, you're trying to chain models, and you have, you know, things you, you want…”

“yes, it's, it's useful to look at the incumbent vendors like Adobe, but you're not fighting really the video editing software spend. You're eating into the production spend.”

“it's going to be five X cheaper to train the same size model the second time around, like, once you've made your mistakes and know what you're doing.”

“the basic idea is to orchestrate LMs in this like iterative loop towards some high level goal. Where they're doing planning, and if memory, and prioritization, reflection, and so you're, you're not necessarily changing the architecture of the LLM itself, but this orchestration allows you to, like, do many new things,…”

“I think of the, um, the core driver as like trustworthiness, right? Citation control of information source. And, and so now you have this architecture where people are using, um, think of it as like traditional information retrieval techniques and search in combination with these models. I think the other sort of…”