The Wisdom Wall

“And then I think another aspect of this is specifically doing this in a way where part of the story is either slowing down China or cutting a deal with China such that China doesn't overtake and, and, and, you know, break this whole proposal.”

“If for many aspects, I think the closed open distinction is almost orthogonal to the safe and safe. People don't understand that, you know, easily because it's easier to do bad mapping than to try to understand the subtlety.”

“Technical moats are not real moats. Like, I, I, there's no portion of basis's long-term terminal value that stems from some, you know, secret RL trick we found that nobody else found.”

“It's not that scale is all you need. You need to also have, have good ideas to, to guide the scaling.”

“You have these crazy, huge systems that have all sorts of interesting phenomena, and, and, you know, if you think about it the right way, they don't grok. There's just this nice continuity.”

“One of the things that is a massive fallacy in AI applications today is, uh, verticalization. As paramount. All of the VCs, a lot of entrepreneurs as well, believe, because it's been true for the last 20 years, that building a very verticalized piece of software is the only answer. And so you've got plenty of startups…”

“one of the sort of fallacies here is a sort of an appeal to authority, which is, you know, well, that person is an AI scientist, so they must know if this is a threat to world peace. No, they don't. They're an AI scientist. They don't know anything more about, they don't know anything more about world peace than any…”

“a lot of the things that we learn with traditional software are like outdated with, with AI.”

“by the time you get that out and it's actually scalable and it can really train it and all the software is ready for it. Uh, and I can now go on AWS and spawn up your new hardware, like, Nvidia will also have been a hundred X faster with their latest and greatest GPU.”

“in my experience, it can often, both of those functions can be a crutch for a broader issue inside of the company.”

“And the key here is you can't automate end to end things with AI, right? And that's why I think like the agentic auto GPT like approach is really challenged because, you know, there's, it's just too hard for AI to kind of reason about every single step, break it down, recursively go in and like, you know, come up with…”

“a lot of them are going through an existential crisis of sorts. It was maybe a lot of things you were trying to solve and work are just going to be solved by training larger models.”

“tweaking the model, finding a fancier algorithm, all this kind of traditional model-centric development that, frankly, we still mostly teach in data science one-on-one courses as what data science is, is effectively dying off.”

“I, I, I kind of think that the whole large scale model thing, It's really a gap in our scientific understanding of these models. Instead of looking at the math and coming up with a better loss function, we figured out that we could throw a compute at it, and get us pretty good results.”

“open core does not work against Amazon because open core, you're building the core open and then manageability, security, reliability. That's your pro pro Amazon is really good at that shit. Like that's, that's the, they know how to do security. So, so it becomes very hard to compete with them.”

“You're actually more likely to churn a customer if they never have a problem with your software. Because they, they look at it and like, why aren't we just using the open source version of this? There's nothing that's wrong with this. We don't need support. What are we paying all this money for?”

“any proprietary software company out there is ripe for disruption by an open source, uh, competitor.”

“So if you hire somebody on race and gender through the urban talent department, instead of through the normal talent department, through the fucking side door instead of through the front door, because you can't see the talent, then you're never going to be good at inclusion.”

“All it ends up doing is you end up having to hire a team of five or six BI engineers to write LookML and maintain these systems for you.”

“So, instead of hundreds of fact and dimension tables, you can just have one time series table centered around a customer.”

“Younger VCs will tend to ignore all those issues and just say, how do I get my hands on the best company? I don't care. Like, and generally they tend to fund people that look like them. Um, I think generally more experienced VCs will have a little more open-mindedness to it.”

“perception is what deep learning does, and actually it only does a small part of perception. There's lots of parts of perception where we use our knowledge about the world that it doesn't capture. And then there are all these other things like common sense and planning, analogy, language, and reasoning, and deep…”

“deep learning is a better ladder for sure, but a better ladder doesn't necessarily get you to the moon.”

“It's actually because we believe that all data is fundamentally time series data.”

“neurons and synapses is not the right abstraction. Models should be programs, specifically programs that simulate the system that generated the data.”

“So if you have to make a career choice to study data science today, I would rethink that. Um, and I'm serious.”

“Um, uh, but I, I do think, uh, hallucinations can be, uh, also very helpful, uh, for AI when you want it to explore novel kinds of proteins.”

“if you end up in a system where basically AI's are running anything, if anyone sort of either puts like sort of secret objectives into that AI or has overt control of those AI's, then they could sort of just directly take over.”

“sandbox is what we've seen this year, and we've seen many examples. They are pretty much easy now for these models to escape from. It's really hard nowadays to say, I'm gonna make a fully, you know, foolproof sandbox. I'm sure it's gonna be resistance against all the coming generation of models. I think we should…”

“not thinking that an agent operating over 10 hours is a black box. It's not. It has a lot, a lot, a lot of data, and you're probably doing a disservice to your customers if you don't understand, like, how it's going about the work.”

“a lot of engineers, they, they, they treat the, the, the code as more precious than the English, when actually the English is more precious because the English affects the performance. The code does not affect the performance, right? If the logic, assuming the logic's the same, it does not affect the performance.”

“In inference in AI, it's the exact opposite. You move a huge amount of data, all the weights, from memory to compute, and you need one calculation to generate the next word. And then you have to do it again. So all the time is dominated by the movement of data. So that's why GPUs have so much trouble being fast.”

“we don't like to make a distinction between Training and infants, because a lot of training is now infants. So when we train a new model, we are generating synthetic data, for example. That's inference. When we train a new model, we are doing post-train, and that's inference. When you train a model, you're doing test…”

“But now that like the app can be built in coded in 20 minutes, like Okay, the long pole is deploying the thing. Um, and so, anyway, vibe coding was easy. Vibe deployment, um, has become like more of the binding constraint.”

“if you accept as the truth that we're going to be running at the limit, then what that means is that the way to get more intelligence is to be more efficient. We can't get more intelligence by applying more force if we're already at the limit. We have to be more thoughtful about how we use what we have.”

“With multi-token prediction, the speed that you get is a function of the accuracy of your model. The more accurate your model is, the faster the inference is, the cheaper the inference is, the more accurate it is. That's not usually how it works, but in this case, that's how it works.”

“if you're asking that question, it probably means you have the wrong co-founders.”

“if I have a developer and I can now give them AI tools and they're 10 times as productive, I'm gonna hire as many developers as I can. If I, you know, have a salesperson and I can give them tools that eliminate the parts of their job that they don't like, which is coming up with account plans and, and doing all of…”

“I think 12 to one is right. And, and the benefit of expanding the number of the average number of direct reports Is it inherently flattens the organization?”

“the old school traditional legacy cloud business was so latency focused because of some of the applications, but this new fleet of AI applications are far less latency sensitive.”

“having the model have a prior of language and being able to like, think in, think in language and then train on top of that, that seems like clearly the right. The right thing to do.”

“the technology is getting so advanced that it makes obsolete the, the, the prior thing that you implemented, which actually means that the rollout takes longer because we have no stable, there's no stable environment to roll things out in.”

“unless the labs build out literally the equivalent of hundreds or thousands of people for every single vertical and every single line of business, that means that there's actually a lot of opportunity in that, in that kind of bridge area of, uh, of the work.”

“even though the, in my mind, everything, the progress is actually pretty continuous, you need to reach this level of reliability. To really make any of these AI tools very useful, and I think we just crossed that probably December last year, at least at OpenAI, that's where I thought we really crossed that threshold,…”

“We, we've seen again and again, the longer the model think for, uh, the better answers we will get. The problem is that this, these curves that we're talking about, um, are not, are definitely not linear, and like they, there's some plateauing effect, and they kind of look, um, look logarithmic, uh, on some, in some…”

“If you have larger models, uh, the amount of thinking time, so the amount of tokens they will think for, um, will usually decrease. And the way that you can think about it is that, um, metaphorically, the model already thinks through its weights when it generates a certain token. Um, so you can, you can decrease the…”

“it seems that after crossing a certain scale of, um, models that know basically everything about the world, and what we call, like, good priors about the world, It seems that reinforcement learning just started to work, and this is not only with LMS. Robotics seems to have, or it seems to be entering the same stage,…”

“when you have hallucination of LMs, if, if, if a model is really bad at saying that it doesn't know, that usually happens in every single domain. You won't have, like, one domain where the model is extremely calibrated about its knowledge, and another domain where it's not.”

“So, so hallucination, um, at least the intuition that people have, um, is that it can come, for example, from, from SFT, and it can come from this, like, pursuing pipeline, but if you have good reinforcement in pipeline, that shouldn't happen too often.”

“right now, actually most models at day zero, if you just drop them in a company, um, arguably they are more useful than most new employees. So they start higher at T zero. Um, but then across time they are mostly constant because they don't really learn kind of company knowledge. Uh, they don't really learn like to be…”

“I think most of the time, the bottleneck is the, the last mile.”

“You can't just sort of trust models to get safer by getting bigger. You have to put in the work to actually make them safer. And, and this is, I think what a lot of AI companies are investing in. This is why we in fact do have models that are improving on these dimensions too, but it's very much not that you get it for…”

“I don't think people have properly internalized the fact that the vast majority of intelligence comes from self training effectively.”

“as we get more and more powerful, I actually think the overhang in the product is bigger than in the model.”

“most of the buttons you add and most of the product services you build are probably more for the human than they are for the model.”

“if someone beats me, Felix, at like building very good products, I suspect it's going to be not because they built a better model, but likely because they figured out a better user experience.”

“as the models get more and more capable, what I'm noticing inside my products and inside my work is that we're sort of like pulling back the edge cases we account for. And I mentioned earlier that memory is just a text file. If Claude needs a database, it will make a database. Those are all arguments against trying to…”

“I think, a thing I tell a lot of my colleagues is that we're really in the silly times of mobile phones, and then if we get really lucky, maybe what we're currently working on is like the Nokia 33 20, like a good phone, but it's not yet the smartphone. It's not yet the iPhone. Someone is going to build the iPhone.”

“pre-training is still the foundation and like, you can never post-train your way out of a week-based model.”

“So because of that, like picking up a lot of knowledge about the word through language is just not really efficient. I don't want to say that it's impossible, but it's not efficient, you know, like to learn about gravity. If you kind of like, you know, have your model train on videos, it's much easier to, to get the…”

“So it turned out to be a really, really good model, but it was like really hard to see that. Wow. You know, I train on images and then like Text perplexity goes down. That was hard to see. You know, like the fact that, you know, you, you train in native model and it's good at, like across all the capabilities is…”

“Not easy to pinpoint like specific things, but again, like, you know, this is just like my personal opinion and, and maybe I have colleagues and, and like the other people like sharing this with me, but, uh, I think we're underestimating how hard, uh, uh, like jagged intelligence is to fix. We're missing how, like,…”

“because if you are Meta or Google, You've got this whole other highly profitable business, which now needs to have LLMs inside it, powering all sorts of capabilities and features, and you probably want them to be your LLMs rather than somebody else's. But you may not necessarily need to make money from the LLM by…”

“But if you've got a bunch of use cases where you need the right answer, as opposed to sort of the right answer, then saying that the model is better doesn't mean anything. I mean, literally, it is literally meaningless. What you're telling me is, I asked the model to compile 50 things, and last month it would get 10 of…”

“the chatbot itself is kind of like trying to differentiate a web browser in that you've got an input box and an output box, and how can you make them different if the whole point is that you can type in anything and get anything out.”

“you start from the technology. You don't, you don't, you don't control the product strategy, which is of course how science works, but you don't know what's going to happen. You don't know what's going to get built. You know, obviously you've got like Sam and, and, and Dario and so on are like setting that fundamental…”

“Most SaaS companies are database wrappers, where somebody realized that here is this problem, and here is the people who have it, and here is a way of turning it 90 degrees, and this is your insertion point, this is how you build it and take it to market.”

“I, I, the, so I don't know what happens, but I do know the harness is really, really important. Like, I think this is the thing that matters.”

“if you want to have a reasoning engine that really truly masters at logic and mathematical reasoning, then you need to somehow get verifiable reward for the proof steps.”

“We still feel like we cannot fully elaborate and emphasize the thing that we are seeing that is the next frontier of AI. That is a generation and verification loop. That is the discovery of verified knowledge.”

“Getting your model to learn about the world from speech, I think it's a terrible idea.”

“What we see in enterprise is rarely things that are solved with agents because that's not necessarily where you would expect, uh, an FDE to be most useful. Where there is more values, value is in, uh, more complex workflows where you will have several, uh, agents interact through a workflow to automate something…”

“and that I think that was the big change in, um, and realization through vibe coding is that agents are good enough at, uh, manipulating file systems that they can use this as a replacement for, uh, their Context window, basically. They can select parts of what they want to read, they can select parts of the tool…”

“there's no real difference between Creating a new thinking trace or calling the right tool. It's, it's all the same to me, because what you're optimizing at the end is what is the best, uh, output for the model to create before it gets results to, uh, to me.”

“getting the data perfect, because we knew, um, this was potentially not the most exciting part of the work, but it was absolutely critical, and any improvement on the data quality would, Tenex, the, uh, improvements that we would get by really, um, improving on the model architecture or things like this”

“Requirements I see for control and governance in enterprise make me think that even if I had, uh, some AGIS model on my, uh, servers right now, if I were to go into a large bank and say, Here is a thing. Please let it control everything for you. They wouldn't be happy to let it do it.”

“you're never going to beat Nvidia at their own game, right? They're going to have the supply chain on lock. They're going to get to the newest memory technology or process technology or whatever packaging technology, whatever it is, sooner than you.”

“pre-training is not dead, but pre-training is boring. So, uh, it's not where the low hanging fruit is anymore.”

“The knowledge is already there in the pre-training, and this just unlocks it. It's just like a step that maybe shows the model how to use its own knowledge, basically.”

“that's where you make the bigger gains rather than scaling the model size. I think that's Uh, one of those things where you will see more progress coming from.”

“Coding agents are general agents. Coding agents can write programs that solve other problems, and code is so general, if there's a digital problem, you could solve it for code, and coding agents make the thing so easy that now you can solve a variety of problems in a way that you couldn't solve before”

“If we let people use agents, they perform very poorly on basic knowledge. And if we let people just do the basic knowledge, they don't know how to use agents and they can't compete. So they can't do useful work in the, in the workforce nowadays.”

“I think at least part of it is probably The models, uh, seeing descriptions of AI, like in the science fiction literature going rogue, uh, and, uh, like, yeah, that probably affects how the models behave in similar scenarios.”

“Industry is really great at executing on ideas, uh, and it's maybe not as good at, like, exploring diverse ideas. Even at the scale of Anthropic OpenAI, uh, there is a lot of focus, uh, in the companies, and there isn't a lot of bandwidth to do exploration, and that has been working extremely well so far.”

“And I think we are at the stage where if we define a benchmark and we can make a relevant RL environment, then we can kind of max it out uh pretty quickly, and so we are going through benchmarks now very, very quickly.”

“I think there are still major, like, compute multipliers, major ways of saving compute that can lead to better performance without just naively scaling.”

“But with a limit on the compute, it's actually very possible to apply deterministic transformations to the data. And create information through that.”

“The other parts are architecture and data innovation. These also play a really, really important part in the Performance of pre-training and probably even more so than, than, than pure scale these days, but scaling is still, uh, an important, an important factor as well.”

“I think what might be happening instead is kind of a shift in paradigm where before we were kind of scaling in the data unlimited regime where, where data would scale as much as you would like. And we're kind of shifting more to a data limited regime, which actually changes a lot of the research and how we think about…”

“So at a fundamental level, you did, you do need the model to know about those things. So you have to train a bit at least on those so that it knows what those things are and knows to stay away from those, right?”

“one point was, of course, the Transformers when it started, but the other point was reasoning models.”

“Pre-training, as I said, I, I think it has reached this upper level of the S-curve in terms of science, but it can scale smoothly. Meaning if you put More compute. You will get better losses if you do things right, which is extremely hard, and that's valuable.”

“using more tokens to think increases your capability, and it increases it, given the computation, way faster than pre-training, right?”

“Pre-training is a little different, right? Because it increases the data together with your increase in model size. So it doesn't necessarily increase generalization. It just uses more knowledge.”

“And within this world, if you think that intelligence is going to become less distinguishable between the companies building it, And becomes a commodity, a commodity probably more like oil or cloud compute than like bread at the bakery, is there's two things that matter, your ability to scale it and the cost at which…”

“So when you take all those margins out, all of a sudden you can start seeing that you can serve your tokens, 2030, 40% cheaper than someone else.”

“But the reason that never got us to AGI is because, well, the internet never actually included the data set of the thoughts and actions that created it. It's the final piece of code, it's the final article you write, but not the thoughts that you had and actions that led to it.”

“what's not going to be dead is the problems that like these AI people don't want to work on because it's so boring to build that software.”

“I definitely think we would keep using pre-training data, not just from an efficiency point of view as well, but also I think there is interesting safety angles, because by pre-training and, you know, all this human knowledge, we're implicitly creating an agent that has similar values as we do, and I think that is…”

“if you're not careful with RL, you can make interpretability harder. For example, one Common thing with modern models is they do reasoning with the chain of thought. You could look at the chain of thoughts to, you know, see what are the model internal thoughts, and then you could also have a thought that, oh, maybe I…”

“Language models do on their own, like fundamental level is they, they are often called as next token prediction machines. And that's not completely accurate in the age of reinforcement learning, but they still operate on mostly on tokens that are mostly text.”

“It is like, yeah, it is some like, uh, risk of, of losing IP, but I think, I think the risk of not doing the right thing and of people not being informed about, about research and not being able to do the best research is much higher in my, in my personal opinion and how, how I, I approach those things.”

“The first thing that is important to know and understand, RL is hard. Like, conceptually, if you think about it, and there's still a lot of depth to it, but, but very conceptually, mathematically speaking, pre-training is dead simple.”

“And like, I don't like in terms of a pure RL, I don't think like really pure RL makes sense. RL needs Pre-training to be successful. And I think pre-training, as I, as I said before, needs RL to be, to be successful as well.”

“The fastest route, or like, the most immediate one is whenever we see a really good blog post where people have, like, done incredible amount of work, um, in an independent fashion, it's one of the highest signal things there is.”

“we think that one of the most important signals of whether or not we are, uh, basically the speed of takeoff, the speed of progress is driven by how much AI is able to assist, uh, AI research.”

“Saying, I don't know, or solving, you know, hallucinations is, ah, intrinsically requires reinforcement learning in many ways.”

“give it math questions, tell it whether it got them right or wrong, and the model will learn. This is, it comes down to a bit of lesson in scale and search, is just allow the model to search, have enough compute to run the experiments, and the model actually ends up figuring out a really effective and sensible…”

“It used to be the case that you pick verticals. I think now you pick principles. And the principles allow you to scale much better than any other previous generation of software.”

“maybe coding is the way that we get to the next level of intelligence. If you call it like AGI or ASI or whatever, the model needs some way to interact with the world and for a model, the natural way is code.”

“Like if you really solve this oracle for organizations just for coding, you've basically built all the capabilities you need to have super intelligence.”

“when you train large language models with reinforcement learning, they become jagged in the sense that they become good at what you wanted them to be good at. And there are some generalization capabilities, but they're much weaker than people think.”

“you can't operate at a hundred X less capital than a frontier lab, but you can operate at, say, 10 X, like an order of magnitude less capital, um, when you're really focused.”

“Malte, among many interesting things, I think one of the most fascinating things that he said was that most of his great engineers who have been with the company for a while, as they've dogfed, uh, you know, vZero, and they've used things like Cursor and Windsurf in-house, is that many of his great engineers are…”

“most people will care, at least in our world, What the smartest buyer at Cursor thought about a given tool than what the smartest person at the, you know, eighty billion software business that IPO'd in, in 2012 might think.”

“LLMs.txt or raw markdown is better for agents. So if you're building developers still today, you have to have that duality in your head. You need to think, okay, How do I make my content, my errors, my developer tools great for agents?”

“If you are doing any kind of visual communication, then there are going to be natural limits to a conversational interface. And, uh, people are going to want to be able to collaborate. They're going to want to be able to control for brand. They're going to want to be able to organize their content. Uh, there are going…”

“There's a challenge for our industry in that we do see kind of slightly bipolar results for these tools in the, in the hands of a senior engineer who can tell good code from bad code. They're very, very powerful, but for graduate engineers, for junior engineers, ah, they can be quite dangerous because they just don't…”

“So this iterative process that the agent does with the help of the tool calls, uh, in all the context makes it so much more powerful than, than a fine-tuned model could, could ever be.”

“It's easy to create a reasoning model. It's dramatically cheaper than pre-training. And so, um, it's accessible. And so there's this huge intelligence uplift that comes for really quite little effort.”

“As agents are rolled out, you'll actually start to see, um, you know, people that are really good at prompting, really good at defining a process, be the best managers, and actually be the best at extending whatever their agenda is in the organization, or making their function the best. And so I think, I think that…”

“right now, it's less around pain, it's less around, like, standard enterprise SaaS cycles, and probably more around FOMO, around missed upside, around value cases that are really hard to define.”

“the only thin, thin GPT wrappers are what you get when you go to chatgpt.com and claude.com and grok and all these others. That's a thin wrapper on a model. Whereas, um, you know, name your vertical enterprise SaaS company. That's not a thin wrapper.”

“the analogy that's been floating around, I think, is, is to, is to compare this with AWS, in the sense that AWS was a sort of an order of magnitude change in how easy you could get a startup out of the door. You didn't need to write all this stuff yourself and buy infrastructure. And so it may be that, like, if nothing…”

“in a sense, like, AWS and Meta are on the same page, and then Meta wants this to be cheap, generic commodity infrastructure that's sold at marginal cost, and they will differentiate on cool Facebooky stuff on top. Um, Amazon want this to be cheap, generic commodity marginal infrastructure, infrastructure that's sold at…”

“There's no viral loop. There's no network effect. There's no reason why you should use the one your friends use. There's no reason this one gets better because everyone else uses it, at least not yet.”

“the difference is the energy is very different where I think most companies I meet with and happen to be in New York and meeting lots of banks. The energy is like absolutely in the category of how many use cases can I, can I apply this to? How do I start to, to lean in more? We've got these 20 experiments running. We'd…”

“this is also where Iceberg or just a data lake strategy or a lake house strategy is not an immediate answer because formats like Iceberg do not understand governance rules, do not understand role-level access control, they don't understand users, they don't understand what does it mean to replicate data”

“Um, and yeah, so I, I, I, to, to, to summarize that, you know, I see intelligence as skill acquisition efficiency. So it's not the fact that you can acquire skills, it's how efficiently You can do it. That's a measure of your intelligence.”

“you have AGI when it's no longer possible, uh, to easily come up with tasks that, you know, you and I can do naturally, but no AI system can do.”

“Their mode is probably not the user experience, but because it's very easy to copy. Anyone can study the product and copy. Their mode is data.”

“we have, uh, kind of an existence proof now that it's actually not that hard to do this, um, and so you don't need to invest all that much in, in data, and you can use synthetic data and, and get a pretty good model out of that”

“one common misconception about fine tuning is a lot of people think that you can inject new knowledge into a model using fine tuning. And that is not true.”

“you can build a very awesome demo on a couple of PDFs and things will probably work. But then you have to scale it up to a million PDFs, and then everything breaks down. And the reason for that is that a lot of these kind of advanced RAG systems still actually don't have this, this Reranker working well enough to…”

“Like you actually can't templateize video because the nature of it is completely, it's unknown, right? It could be saying anything. It could be doing anything. Anything would be happening. The, template just can't fit with, it can't understand what's happening in the video until now, obviously.”

“I think hallucinations in text obviously can be a bad thing, potentially, because they're saying stuff that, you know, isn't true, but in a video, like, you are delegating a lot of decisions to the model, right?”

“So, like, you know, the, in other words, like, customers weren't opening these emails because they didn't believe that the email could be helpful if it came so instantly, right? So it was like an eight-minute delay that a lot of our customers use.”

“What we found is reasoning over long contacts is actually not very good. It's like, not good at all. Like, once you cross 32,000, uh, tokens, that performance of reasoning, you know, just goes down to hell, like, very, very quickly.”

“What, what models are still are today is they're like, they're sort of like completion engines, right? That's, that's how LMs are trained. The, the sort of auto aggressive models where they try to predict the next token. They're still next token prediction machines. Um, reasoning is, is, is different.”

“I believe, gradually over time, enterprise software technology, at least infrastructure technology, wherever there are two things that are mostly the same, and one is open and one is not, the open is going to win over time.”

“So, so I do think that post-training is what makes this, like, really big lab models are, like, different.”

“Um, so, so I do think that evaluation is the biggest bottleneck for AI adoptions, because unless, like, if we can, like, if we can, like, develop a more reliable way to evaluate the application, that application is not going to get adopted. Like, or, or maybe, maybe it can. Maybe we need some billionaires to just,…”

“when we ask humans to generate like what they consider the best plan for an actions, for a task, it's actually like not quite the best plan for AI, because what is, uh, what is easy or efficient for humans is not the same as easy and efficient for AI, right?”

“none of this is about productivity, right? You being able to go through Three decades of consumer research on Listerine to be able to develop a new flavor, right? You being able to, uh, develop all of the go to market, uh, around that new product in literally a third of the time for a third of the cost. Like, none of…”

“tech people in SF obviously live in a bubble, and it's kind of, it's quite techno-utopist, and I think they underestimate how, how much human inertia there is to the diffusion and adoption of technology in, like, the real world.”

“So it's sort of always felt to me like, you know, in, in hindsight, it looks like the right idea, but applied to the wrong problems. And I, and I always like wonder, you know, now, like I'm looking at like what we built at modal. Like I I'm kind of convinced that like serverless is actually makes a lot more sense for…”

“one trap that I think a lot of startups fall into is that, you know, you go out and talk to a lot of customers and, and they sort of insist on, on self-hosting. In my opinion, like, you know, sometimes that's actually like a true, like hard concerns, but, but in many cases it's actually more of a soft concern. And, and…”

“if GPU prices were to crash, like the relative value of that software actually goes up. So like, I'm not necessarily sure that like a crash in GPU would be bad for us. I think it actually could be great for us because, you know, if, if GPU prices go down, there's going to be more applications using GPUs.”

“there's just this huge tax that you pay to, um, to have to build a distributed system that scales out and can do, like, you know, distributed transactions and shuffling data and, um, and, you know, if you were gonna design something for kind of modern hardware, you could make it, Much, much faster. Um, and you could…”

“If you think about it from first principles, there is nothing you can do to be 10 X better.”

“So for new products and new companies, I would argue, month to month is great. Um, because your customers can try at any time, uh, which means you'll get the, the hard, uh, reality to hit you in the face and you can't ignore it, you know, which is a problem when you have, so you say you sell one year, three year deals.…”

“it works great for some use cases, but you need to handhold the users a lot more. Like, users don't necessarily know exactly what to ask for and how to ask for it, or what to go next, you know, once they've asked a question.”

“today when people are running RAG or prompt engineering, um, those are, those are search. That's not AI. Um, it's like keeping the AI frozen and fixed.”

“if you can enable your direct sales team to kind of handle a lead from cradle to grave, uh, I think you're going to drive efficiency as part of your sales organization. I think you're going to improve your customers buying experience and you're going to have higher retention and higher expansion”

“we believe that encoder-decoder models are better for when you deal with semi-structured documents than decoder-only models.”

“and I, I actually think now the bottleneck to their, um, to their application is UX. It's actually giving people the tools to very easily deploy those models into not just a few Very central core use cases within a large company, but actually into the fractal of every single function, every group within an enterprise.”

“the fact that no code by definition means that the output of the Turing bot, the AI builder, Is actually in a form that a non-technical user can understand. So the very person who's asking the AI to generate the app can then understand fully its outputs.”

“In fact, my thesis is you could freeze all model development in its current state and say, you know what? GBD five is never going to come out. Like there's not going to be any better models. The models we have are all we're going to get for the next 10 years. I think we would still have 10,000 times more value creation…”

“Um, you know, which is not to say you have to be PLG to start with, but I think every PLG company, and this is true, whether you started in Zerp or after, um, I don't think it's going away, but I think if you start with PLG, you will eventually run into an asymptote of PLG driven growth alone.”

“there's probably some pricing arbitrage in other categories where people aren't paying attention.”

“a lot of those companies, uh, to put it bluntly, uh, don't have a lot of traction yet, because, uh, it turns out that, if you're trying to do something, and I can, I'm not picking on them, but like evaluation or monitoring and all the things, well, you need to have LLMs to monitor, right? So you need, your customers…”

“Do you really need a, you know, a specialized database to do that? Or in a context where some of the more general purpose players like the MongoDBs of the world have started announcing vector capabilities, you know, is that good enough? The answer to that question will be enormously impactful to how successful those…”

“a lot of the learnings in AI are not on the technology side. It's not about creating a new AI model from scratch. It's about learning how to prepare data, how to augment the data so that you don't need quite as much of it, learning how to normalize or regularize the data so it all sort of fits into a standardized…”

“basically every enterprise software company for the sake of argument is unbundling Oracle, Gmail, or Excel.”

“you don't look at this and say, well obviously the solution is that we needed to have a database regulator that makes sure that databases don't have bugs. Um, rather you look at it and say, well, this is an institutional failure. A in Fujitsu in the post office and B in the legal system, not properly testing the…”

“It's because cold outbound doesn't generate demand. Cold outbound generates awareness manually, uh, among, you know, otherwise highly targeted folks.”

“I think radiologists will not be replaced by AI. They will be replaced by radiologists using AI.”

“I have learned the hard way that it's not that transferable. You actually need, if you're trying to push the state of the art, you need people who have proven experience in the very, very specific narrow scope of problem that you're trying to solve.”

“We shouldn't be approaching AI from just simply, how can we automate a task? Or how can we automate a little step? Or like, you know, how can we summarize this paragraph? Or you have to actually look at how can we automate the outcome?”

“The era of SaaS is an era of like right click view source. You know, you know, ultimately like if, if there's a single thing you've done, other people can do it. The job is to build a brand around having the best product. And to do that, you often, you have to have the freshest tech, the best designed tech, and you…”

“when you talk to the folks at the hyperscalers, they will say, we are, we're, we're like making sure that our solution is like covers all these different areas, but it's mostly not. Because we want to compete in these areas. It's mostly because people come to us with RFPs and we need to be able to check all these…”

“Had to build, you know, there's like the iceberg, which is 10% above the water and you can see it. And then there's 90% below the water for a, for a data company. The platform is the below the water part. And then the functionality that you build on that platform is the 10% of the top. And a lot of times different…”

“If what you were trying to do is maximize the multiple that your software company was trading at, then AI is the enemy because all the multiple expansion has moved from modern data stack to AI.”

“That is something that is, works better than, um, uh, fine-tuning, for example, because if you fine-tune, then you're still dealing with potential hallucination Fair enough, with RAC that's possible too, but it's like, it's less, um, uh, uh, uh, it's less risky.”

“Just hiring a lot of salespeople, right? It's not a go-to-market strategy. And that's what a lot of startups did, right? Hey, we have just got a hundred million dollars in funding. We're going to hire tons of salespeople. They know how to do this. Um, good luck. Uh, that's not going to work.”

“if you didn't sell the first million or two in revenue yourself, Don't expect anybody else to do it, right?”

“And it's that RL loop that is very interesting because since it's programmatic, since we have an Oracle of truth, we can scale this up far larger, right? Magnitudes larger than what you can do with human feedback today.”

“one of the most important insights was, and maybe as important as building a great family of models. And I can talk more about Firefly and the underlying models and how we trained it and how we didn't train it and everything else, but was simply the insight of what we call the context bar that shows up by default in…”

“we're never going to catch the bad actors because anyone can, you know, make anything and it's a cat and mouse game to have algorithms that tell whether something was edited or not.”

“Which is that why can't it be another LLM or a pipeline of LLMs that can help with that feedback? Um, I think manual labeling is very tedious, especially for our target user, which is a software engineer. Um, and I don't think people should necessarily have to do all that if zero shot, uh, large language models are…”

“the bigger companies, like Databricks or Snowflake might build these tools and offer it on their sequel editor or, like, some kind of, like, you know, assistant And that could probably solve the problem. It's not a great problem for a startup to work on.”

“vertical applications give you the best chance at creating a longer term competitive mode”

“there are definitely some that have, you know, tens of thousands of stars on GitHub. Um, but if you look at the use, where those stars are coming from, it's mostly hobbyists, and I'm sure you can monetize hobbyists, um, and sort of grow from there. But I think it is like a very big stepping stone for these AI companies…”

“context size and model quality makes it such that an assistant that knows it all within a company is, is, is, is still kind of a bit farfetched because it'll get confused if it sees too much information.”

“if you want to build a strong AI research team today, it'll be 10 times easier to do it in Paris than it is to do it in SF, uh, with OpenAI, uh, as a lab, as a competitive lab to, in the same hiring markets.”

“if your metric is smooth, uh, you actually see slow performance improvement over time, and if your metric is relatively discrete or not smooth, that's where you see the emergence, and it's actually more about the evaluation metric Than about the emergence of the capabilities.”

“the trajectory of issues and concerns that people are running into are similar to, you know, like the, the path is similar to what it was in self-driving, which is, you know, How do I get like this, you know, runtime safety? How do I get like runtime constraints, et cetera?”

“existing, for example, like model risk management frameworks don't really apply when you haven't built the model yourself. You know, you didn't like curate the data that the model was trained on. And so you can't make any claims to that.”

“at the end of the day, these models are like next token predictors, which is, you know, like they kind of look at like what they've predicted until now, and then, you know, figure out like what the next token they're on is. Um, and from that, like, even with fine-tuning, that fundamental behavior, uh, remains the same,…”

“LLMs and Generative AI really helps solve, like, um, the first mile problem in ML, right? Uh, but the last mile problem, which is, like, how do you take this generic generalizable technology and make it work, like, specifically for your use cases, uh, and for your actual application, like you need other more…”

“Sometimes you can over index on the technical and getting maybe, you know, your platform to be five percent more fast or supported on one more browser might not 10 X you the way that spending that time building a community would.”

“And yeah, I don't think you should attempt it unless you have, you know, hundreds of millions of, of responses, um, that to really like train your own model, um, versus just fine-tune it a little bit. Um, like certainly not from scratch.”

“I don't think they really, I think people tell this, the story of sort of the silver bullet. Of like, um, you know, the eager execution model. But I think the reality is they just built a product with so much more empathy.”

“if the web made the cost of distribution essentially free for businesses, what AI is doing now is AI is making the cost of cognition essentially free.”

“these types of videos Essentially became not a replacement for video production, but a replacement for text.”

“Um, and I think there's lots of other things, uh, like this, but I really think that one of the biggest opportunities right now is actually not necessarily selling the tools. It's actually creating the content and monetizing that content. Um, because the tools are becoming so good now that, that this becomes very, very…”

“Enterprise data knowledge is the durable moat in AI, I'd argue the only durable one”

“I actually disagree with that. I think like we are learning so much literally day by day. I mean, GPT four was released yesterday. Um, and there is possibly, you know, a second mover advantage. There, there, there, there will be lessons that we learn again from kind of seeing this first wave of startups, uh, emerge,…”

“And I think that that is the promise of LLMs and startups. It's, it's not to build better versions of things that exist today, but rather to kind of reimagine some of the tools and systems that we use, um, in an LLM centric way.”

“most companies do not in fact have like big data. Perhaps there are a few companies like in genomics or like astrophysics that like actually have, you know, Petabytes of data, but like most data sets are not that big.”

“open source ate the software world, and now I think cloud services are very much cannibalizing the, um, the open source business model.”

“the mandate to open source is not nearly as strong as it was when we started Cockroach.”

“as soon as that person starts to figure out what that motion looks like, you're probably going to need to replace them, because the person that can figure that out is not usually the person that can Mentor other salespeople and start to scale an organization and really codify that motion into something that can be…”

“What they're worried about is the vendor lock-in from the hyperscale cloud vendors. They're very worried about that. So the right way to assuage their concerns isn't so much to convince them that they're not going to be locked into your system. It's to convince them that if they use your system, they're not going to be…”

“Centralization of the data, centralization of the organization, functional division between the data and non-data, those are the things that have led to kind of a system that is fragile to scale and change at the macro level, not at the bits and bytes level, right?”

“You don't get to pick your channel. You can, you don't get to say, oh, I want my ASP to be 50 K or 60. That's not your choice. You have a product. You have a market. If it has fit, you have to find the right channel to connect those two.”

“We're going to host open source software in the cloud, but the difference is we'll create open source software. That way we get competitive advantage with respect to anyone else who would want to do the same thing, right? Otherwise anyone can pick up any open source software and host it in the cloud.”

“So you actually, or chart wise should separate those out because otherwise what happens when you're successful is that the former gets all of the resources because the big enterprises have infinite, uh, demand for your, uh, for the things that you're doing.”

“the best way to monetize open source is open source as a service.”

“So a lot of American, American internet companies are like a lot older than the average, like the new, like Chinese internet company. So it means it's like American internet companies have legacy systems that you from like, 20 years ago. And just have to build the system on top of it, whereas either Chinese can be a…”

“to achieve that, you need to build a developer tooling company, not a security company.”

“a security person needs breadth. They need, if you, if you solve the security threat, but you only solve it for 50% of my apps, you're not very helpful security. I'm going to need to buy another tool or find another solution for the other 50% because I need to cover against this threat across 80%, 90% of my stack.”

“the industry has been seeing containers as the evolution of the VM. And they've been trying to retrofit VM patching practices onto containers while we saw containers as the evolution of the app.”

“it became clear to me and, and, and to many people that, um, that to have, uh, more of a commercial engine behind Arrow and the Arrow ecosystem, uh, was important for, uh, enabling the ecosystem to continue to grow for us to be able to pour a lot more resources into the open source project.”

“we believe that, uh, we've seen a 10 year acceleration in digital transformation in our market easily.”

“And the really critical thing we did that laid the foundation for our eventual success was, uh, that we said that it was our responsibility to make the data match. Which is actually unusual in the field of data integration. Most data integration tools, they see themselves as like a platform, right? So they give you all…”

“no matter how capital efficient you are, you end up the, the faster you grow, the sooner you need to raise again, unless you want to just sit there and have, you know, one month's payroll in the bank account, which I don't think you want to when you have a lot of employees. So it is this funny paradox of fundraising,…”

“Some entrepreneurs look at this as a big tech partnership as this silver bullet that's going to solve their go-to-market problem. That's not going to happen, right? It's going to be a very long journey and going to take a long time, require a lot of, you know, align incentives.”

“human intelligence is no, it's not general. It's actually very, very specific to our world. It's very biased. Uh, it's really customized to survival human in, on our planet. So the concept of general intelligence, I mean, yes, humans tend to have a more general intelligence than, than the computers today, but there's…”

“if you have a, uh, founder who's, like, say, telling you, like, some pretty aggressive, not true things in the pitch, then, you know, that's gonna be a tricky founder to work for, because it's very rare to have a person who, you know, lies only to investors and not to their employees and that kind of thing”

“it's, it's very hard for what we do, uh, to lead with, you know, AI, uh, because you, you're bound to disappoint, because the generic case is very, very hard to solve”

“when you ask, when we first visited customers, Um, to figure out what they wanted from our product. When you ask them to, to pick between a false positive and a false negative, um, they'll say, oh, give me the false positive, I'll decide if it's right, but I want to know. It's not true. Uh, you give them two false…”

“we're really moving from, from a world where, uh, only humans could do natural language, which is why you have search large, uh, customer support, sales, community, communication departments in companies.”

“we believe that actually if, uh, all the other companies than Facebook, Google, Apple, Amazon, uh, are actually all contributing to the same open source, uh, technology, they can actually end up with something way better, uh, than each one of these guys, uh, could build.”

“So that sort of obvious inference is something that we as, as humans do effortlessly, and that if you just have a box, if you have an autonomous vehicle that's just, just, you know, rendering everything as moving physics objects in the world, Is difficult verging on impossible?”

“So to put it, to put it this way, we, I don't think we are ready yet, uh, or anyone are ready yet to sell to hedge, hedge funds that want to use this data to predict quarterly earnings for stock listed companies. But if you want to understand retail analytics and real estate, understand like trends over time, It is…”

“So the community is still very important, but it's gone from basically substituting for your engineering org to substituting for your product management organization.”

“data science is not actually a job or a role. It's a literacy”

“point-and-click graphical tools are really very impoverished when it comes to expressing the full plethora of opportunities and possibilities. And what we, in fact, see is a lot of the graphical tools for, you know, advanced BI++ kinds of tools, they ultimately kind of degenerate into a pile of complexity.”

“data privacy and anonymity and the, kind of, the idea that we can live private lives, that is contingent, it's kind of a herd immunity, right? So even if you turn off all your tracking stuff and you're a super, like, crypto nerd about all this stuff, if everyone else around you isn't, You're a glaring hole in the data,…”

“you cannot buy off the shelf ethical AI. You can buy effective AI, but if you're not doing ethical things as a business, you're not going to have ethical AI.”

“The hardest problem in AI is coming up with a representation. Once you've done that, the rest follows, and when you don't do that, you don't really get very far”

“AI can be a great tool for democracy. It is also unfortunately an amazing tool for authoritarianism.”

“The real danger of AI is not that computers will get too smart and take over the world. The real danger is that computers are too stupid, and they've already taken over the world, right? Computers are making decisions about us the whole time, and they're dumb, right? Dumb AI is vastly more dangerous than smart AI.”

“if a typical person can do a mental task with less than one second of thought, and we can gather an enormous amount of data, of directly relevant data, we have a fighting chance. So long as the test data aren't too terribly different from the training data, and the domain doesn't change too much over time.”

“deep learning doesn't even really fully understand the relations between parts and wholes. Um, it certainly doesn't understand what a silhouette is, and it gets it wrong.”

“deep learning can't do what we call compositionality. It can't put ideas together.”

“There are a lot of different components to cognition, and so we have to have a hybrid model. There's just no way that one system is going to do all of these different things”

“Uh, if you want to find these folks, there's a really good place to look, uh, which is hiring social scientists. So social scientists are trained to think this way. Uh, they are analytical thinkers without some of the, the technical and quantitative background that a lot of STEM folks have. Uh, but they're much cheaper…”

“I think this is probably the number one error technical founders make, and it's the following, right? So, alright, so I'm in my lab, and I make my cool thing, my cool widget, whatever it is, and then the big logical fallacy we all make is that whatever, that widget, that thing we've made, actually has intrinsic value.”

“especially in enterprise, there is no single decision that you make that's more important or directly tied to your value than pricing.”

“In my experience, when you hit about 30 million-ish dollars a year, you actually have enough of a market to actually incent the partners.”

“Listen, if you're in the first hundred million dollars Because of revenue, and you're the innovator, and you own the code, I think it's incredibly difficult for these These larger incumbents, you know, Google, Amazon, and Microsoft to compete against with you. They can't build a sales force that will, that will…”

“so right now, actually, net dollar retention to investors is probably more important than how much you can get on the first, you know, dollars.”

“Now, number one, do not pretend to be a human.”

“And I think there's probably even more value in whatever agent that you go engineer to over invest in the reasoning engine.”

“But those excitements Turned, turned out to be premature because, um, it's, yeah, game playing is, um, an easier problem compared to real world problems. Those, those techniques do not translate to real world problems.”

“If you market it in enormous quantities to speculator types, and you give them big discounts, you put billboards in Times Square, leading people to believe that there is a group that they're relying on for a return of investment, that's going to be considered a security. But if you don't do those things, if you…”

“And in that context, we own our route. Nobody can take that away from us. We can be forced out of our country with no possessions, and we can reestablish that in another country when we get back on our feet.”

“you can sort of, like, you know, you can download TensorFlow, download some data sets, and over the weekend, probably come up with, you know, if you're a good programmer, come up with something that can do, like, 80% accuracy of whatever, let's say OCR or something, right? Um, but then the last 20% takes you, you know,…”

“the way, the way I think of crypto networks, as I call them, is, is they are, um, digital services, and that's, I mean, in the broadest sense of any kind of digital service that are owned and operated by communities as opposed to being owned and operated by companies.”

“what, what tends to happen with these big platforms is they start off being, uh, very open to, kind of, third parties to, to, to startups. Like, if you look at Facebook early on, they, they embraced Zynga, and they embraced media companies and things like this, and then as they got bigger, um, they started to fight…”

“the thing that's worked for me as an investor is to not, to, to not try to outsmart, ah, really smart developers. Just sort of let them show me where to go and follow them.”

“speed is more important than quality, which is more important than optimization.”

“all data science And all AI and all ML is fundamentally vertically specific. Which is the dirty secret about AI and ML powered businesses. They are fundamentally vertical problems.”

“AI and ML has zero value in marketing.”

“Even if that data is preserved forever, um, no one will be able to amass the same dataset, um, if you're the first Uh, mover into that space.”

“there is no way that institutions are actually going to get into this space until the compliance team actually leads from the front.”

“And some of those customers will ask you to do an on-prem private deployment, will ask you to not use their data to train anything that you do, and you need to reject those customers. Um, because you are not going to build a company that has any sort of moat around it, that has the world's best model, unless you get…”

“In that world, even the most advanced models, deep learning, and machine learning frameworks can't beat them because they have access to this kind of core underlying data.”

“One of the, one of the things that, you know, operators view as the holy grail is permanent capital, right? If I, if I don't feel like I need to sell this building, why should I, right? And the only way you do that is if you actually create a secondary market, right, where essentially a stock exchange where people can…”

“our belief is that, um, AI and machine learning, you will not win this war by having the better algorithm. The differentiator, the value proposition, is not in the algorithm. It's actually in the data.”

“deep learning requires a lot of data, relatively speaking, and therefore, by definition, cannot detect recent changes in the data, emerging patterns, ah, and therefore, it's, ah, really, really, ah, inadequate to predict human behavior that is, ah, actually dominated by those emerging patterns and recent changes.”

“basically my advice is not to start a startup with a data scientist. You know, focus on getting to MVP, getting some traction, generating some real data, make good decisions early on, and then hire data scientists as you scale.”

“Do you think you're gonna have a fundamental algorithmic advantage in perpetuity or not? And in the vast majority of cases, um, the answer is almost demonstrably no.”

“if you're really, really, really honest, it's finding the 10 X teams, and you can kind of figure out the markets later, and so that's not like a very tweetable line, but I think it's the truth.”

“what we learned is that, ah, stream processing is in fact much more than a faster MapReduce layer. It, it, in fact, most of the applications that do stream processing look much more like a microservice of an application, and less like a faster version of a batch or MapReduce job.”

“Because if you're in that meeting, and the entrepreneur makes you feel safe and calm, and, and, ah, you know, kind of, ah, very much in the linear mindset, ah, they're not likely to, you know, profoundly change the world.”

“we had to actually follow, in many ways, follow the instincts of the best people of their generation, because they're chasing the things that have the highest rate of change, that are at the, at the intersection of a collision of different sectors and segments”

“doing open core on a developer tool is not going to get you very far.”

“if you try and monetize and constrain adoption, you're just gonna open the door for another company to come in and, and, and compete with you”

“Uh, and, and machine learning is the game of kings.”

“unless there's something proprietary in the open source, which by definition it's sort of already out there and open, um, I think it's very difficult for a license to protect you from Amazon coming up with good enough in a competing service.”

“But it's another thing if you import those same techniques into robots. If you have your robots doing reinforcement learning, which is basically trial and error learning, they start knocking over the furniture a 100,000 times, you're probably gonna send it back to the manufacturer.”

“Um, oh god, yeah, um, so big companies will come to you. If you're a startup here, they'll come to you and say, hey, we don't have any money, but we want to do a partnership, and like, they'll use terms like joint value creation. Like, you should just run fast from these things. They're just always a waste of time.”

“I think that it would be a mistake for us or for anyone like us to build something that's closed source in terms of a general OLTP database system.”

“I like to use Steve Blank's definition of a startup, that a startup is a, is a temporary organization in search of a scalable and repeatable business model. And in that sense, every publisher is now a startup, because the business model of publishing just completely evaporated, 2004 through 2008.”

“Although this is kind of conducive to fast-paced work, these data-first or method-first approaches can actually amplify issues related to bias, fairness, and inclusion of minorities.”

“Because we've got some management monitoring administrative tools That the big, well-capitalized vendors can't simply pick up and use against us. We've got a reason for customers to come and talk to us. It's not really about lock-in. It's about lock-out for IBM and Pivotal and Friends.”

“I am convinced that Only a hybrid company can basically generate the revenues and the margins that allow us to invest forward in innovation in the platform.”

“if you're just counting and you tell the, tell the, the bureaucracy that, that the numbers are important, and even if you don't explicitly say that, but implicitly, if, if, if you have folks, uh, in the minute, senior officials that keep using higher and higher numbers, it encourages people to chop it up. And that's…”

“Nobody was looking at nobody before, and now everybody is looking at everybody all the time. And therefore, manipulation is on the rise.”

“what we learned there was like, you know, that sounds great, but actually what you need to do is just book some bad loans. Um, because you learn a lot from the loan, you know, you just make some loans, and then as you make some bad loans, you learn a lot. So, again, most valuable data was once you actually had real…”

“So that's an excellent question, and, ah, there is definitely an, there, ah, you know, there is an inflection point where people, ah, think that, ah, the total cost of ownership, ah, you know, once you're, ah, so suppose you're a Facebook skill, right, of course it doesn't make any sense to do it on the cloud.”

“Great question. And so, again, this is biased predominantly towards sort of tech companies, but I think the biggest one is actually interviewing them as if they were software engineers. And so, in particular, the, the fellows, when they graduate, the program go into interviews, and a big portion, sometimes even the…”

“The only defensibility that you can get with data is defensibility around your own data. If you're getting data from somebody else, it's not defensible, because they can give it to somebody else as well.”

“shows the problem of thinking that software is any kind of defensible advantage. I don't think it is. But if there was something about The data, ah, and the network around that software, um, that made it so that it wouldn't be easy to leave the first company and go to the second company and leave the second company and…”

“I, I, I, I don't think selling the data is the right thing to do. I think the right thing to do is to build services on your platform that take advantage of the data that let the people who might buy the data from you instead come and build businesses And transact on your platform.”

“in consumer internet, even if, if big data and machine learning is going to be the thing that ultimately locks in your value proposition for the long haul, I don't think you, I don't think you get into the market and, and, and get the scale position that allows you to access that data, um, with data science. You do it…”

“when entrepreneurs come in and say, We're gonna bootstrap, um, you know, one side of our market, the demand side or the supply side, by going to institutions or going, you know, you know, kind of bypassing the, the little guy and going to large chunks of supply or demand. That always strikes me as a bad idea. And I…”

“it has to be something that they desperately need, and it has to be an order of magnitude better if you're a small company. You can't just come in and have a slightly better solution than IBM and then somehow win an RFP. It just, it never works.”

“won't speak for the, the large companies or people, um, who are working on very specific problems, but when you're generally building systems that, um, in a startup environment, it's really important to understand why your system is doing the thing it's doing, or else it's going to change what it's doing, and you're…”

“data science products should receive no special treatment. If a designer Can do a better job solving a problem by changing the color of a form than I can do with some AI model or something, then that design product should win.”

“And the real, what we found is that the differentiation comes from old-fashioned, the ability of their people to use it. It's in the feedback loop, which is truly proprietary to them. And as the algorithms improve, based on the, that's where the competitive transition comes from.”

“if you make a thing, any price advantage you have is temporary because somebody will make it faster and cheaper, usually coming from overseas.”

“when mathematicians, um, ask me how do you become a data scientist, I give them a lot of advice, but one piece of advice I have is if you interview with a business who thinks that you're an implementer and not a business person, then don't take that job.”

“I mean, um, I think, I mean, it's, you can't do analytics in key value.”

“the verticals where people die, or vast amounts of money are lost, if the right answer is not arrived at in increasingly smaller windows of time, are, in fact, the most aggressive customers for some of these technologies.”

“pretty good tech with unbelievably good design actually will probably beat great tech with so-so design, and will definitely beat great tech with crappy design.”

“So I, I actually think privacy is not a threat. It's a business opportunity.”

“And my, my theory here, uh, after thinking about this for quite some time is largely dependent on the elasticity of the demand of the product, uh, when its prices go down.”

“there may be some intermediate period, an intermediate period that I would say we're currently in, where the AIs are maybe capable enough to cause at least moderate problems, and then I think increasingly able to cause quite large problems, but they're not necessarily so capable that if we, like, you know, tried quite…”

“where like an obvious example would be if the AIs are thinking mostly in activations rather than in words, that seems very concerning because our ability to oversee activations is much worse.”

“And it's not because the model is dumb, but I think it's because probably, I mean, part of it is because of the training process and how they are trained to be efficient, how they use their token. But this means that they start to, uh, and bundle a lot of semantics in some token. And generally it's just seems like it's…”

“in OpenSoup Mobile, you can download it. Nobody can, like no country could take it out from you once you download it. You can find it yourself. If you operate it on, on your data center, like local ground data center, I feel, I feel like you start to have the beginning of a sovereign stack.”

“Speaking is just so much faster than writing things down, and in fact, when you try to write things down, you are Actually, essentially, uh, trying to summarize all the crazy thoughts in your head, and so that's why it takes a lot of time. And it's useful for your eye, because it's rude if somebody just blabbered and…”

“And then I think after a one, it was oh three, because I think oh three helped prove that not only could you scale the amount of reasoning at inference time, but with better training, with more compute, better data, et cetera, in the post-training phase, you could make the reasoning higher quality, more efficient, and,…”

“At the end of the day, coding is subjective. It's an art, uh, and you're not going to solve an art through verifiable rewards.”

“humans are already used to working with non-deterministic systems. It's just the systems are normally their coworkers, not their computers. And in many ways, like companies and processes, it's all about how do you design a system for non-deterministic entities, i.e. like humans, to coordinate together to solve a…”

“I think the framework I like is thinking about it as training data. Except you're just training the model at runtime. It is training data. And because you, the model's learning at inference time, the total amount of training data is far lower, right? Like the whole amount of context in your system that the agent would…”

“as models are getting cheaper and cheaper, more and more of that actually can just be done, um, using inference instead of using determined, like things like graphs or things like embeddings.”

“I, um, I think law actually is kind of like that. You know, in many ways I think about, I think maybe the founding fathers would have been really good, um, context engineers or agent managers, because you had to, you had to write, you know, a piece of English that was going to be, you know, interpreted at runtime.…”

“These are not the tokens you're going to find online. Like you can't crawl Reddit and find out about what happened on a construction site, right? Nor can you do, uh, you know, test time reasoning about it. You can't just, you, you can simulate all kinds of environments, but Really what our customers need to know is…”

“The right metric is tokens per second per user. That that's how fast you get the first token all the way through the last token in, in your, In your response.”

“the way to think about scaling and, and say, scaling laws is not small to big, but big to small.”

“The code base has so much of the context in coding. Whereas in the rest of knowledge work, the context lives across like 20 different things. Some digital and some very not digital, you know, kind of mediums.”

“once you start having models that are really good, you accelerate yourself. Um, especially in terms of coding, given that we all code internally, uh, yeah, you accelerate yourself both for having these models, like train the other models, but also like build like the tooling that we need as researchers to like do our…”

“So we really have like different sub teams, uh, including pre-training and you, you have like the mid training stage and like you have some post training and usually the closer you get to, to products like pushing being the last one, the faster the iteration cycle is. Um, and if you're more upstream, the slower the…”

“I always thought That it would really help, ah, kind of your reasoning abilities if you have a lot of multimodal data, um, And I still think this, but, but for example, like if you look at entropic models, they tend to not be that good on multimodal, and they are still really smart. Um, so it seems that, uh, it's not…”

“When we are training more agentic systems, you only know whether you're correct at the end of your very long rollout. Um, so you get very little information per token of whether you were correct or not. And it's hard to say, uh, it's, it's hard to basically do attribution. It's hard to say what part of your entire…”

“as we get, like, better models, uh, we have this self-reinforcing loop, and we have this, this, like, capability flywheel, where better models become better teachers for other models.”

“the way people usually run these things and cloud code or open claw is usually on their own computer because it helps them, you know, organize emails, you know, search whatever documents they have and whatnot. There there's a one, there's a high security risk there”

“the code anywhere product had no enterprise use case value. It was all single developer value and single developers do not want to pay for these things.”

“A conference is it's entertainment first and foremost.”

“I, I, I think the way you solve things is through, through ongoing exploration of, of what's happening and through, through interaction with the frontier.”

“Reasoning models were much more effective because you can't really do the same trick of optimizing for a probability with a reasoning model that has a whole trace of reasoning that happens in the middle and kind of reflect a bit more. So it's much harder to break reasoning models in the same way.”

“Coding agents are extremely good Mechinterp researchers.”

“basically progress happens when The current crop of young researchers ignores the things they've been taught that the, the old guard believes.”

“I don't think we should teach people that they should trust a singular company with all of their passwords.”

“model collapse mainly happens when you have a loop that is Completely closed. Right. And, um, if you don't have any outside signal and just the model, for example, talking to itself or, or operating in a, in a very, uh, like a, like a restricted environment, um, there's a good chance that your model can access, but,…”

“short term, um, I would say like building a specialist model is like probably the fastest way to learn like what is actually possible. And, um, in, in many cases, these like specialized model are becoming, um, Stepping stone toward a generalist model, which, which is like super valuable, right?”

“in mixture of experts, you have flops free parameters. So, so parameters that they're not actually bringing any flops. And in, in like looping, you have, um, uh, parameter free flops where you don't have extra parameters for the extra flops that you're throwing on this. So it goes the other direction of the sparsity…”

“there was also benefit of doing that simply because the rest of the, that the machine learning field, which was working on, on, on language, they were using this, like, uh, like architecture. So they were building infra for it, making it faster. And, and, you know, like, like the, sometimes the hardware is kind of like…”

“But if you have incremental generation, so if you have text and then an image and text and image, you can get your model to generate these details one by one. So you, you never expect your model to, to generate an image, a perfect image in the first shot, right? So, so you expect model, your model to plan about this…”

“people don't experience average performance of these models. They experience, like, the failures. If you have your model doing a dumb mistake, the damage in the trust that it makes is, like, bigger than, than, like, you know, the benefit of getting hundred things right, right?”

“And we're still at the stage of, you know, taking a PDF of your catalog and putting it in your company website, um, as we try and work out what we should do with AI.”

“tell that next time you hear a software developer saying, like, AI is a completely different thing, and nobody has ever abstracted software like this before, like, yeah, we've been doing this for 30 years, 50 years.”

“Well, you know, everyone in, in, generally in a bubble, everybody's a rational actor. Almost everyone's a rational actor given their situation.”

“The hard part of writing software is not writing code. It's all the other stuff around, like, what, what should the code be doing? And how would we tell people that they should be using it? And what should we charge? And how do we go to market? And which bit of the market should we be selling to? It's all the other…”

“I, I think a lot of the differentiation is in like the instructions and the tools and the skills and that basically, yeah, knowledge of how to do a process that you encode into natural language and give the agent and then the tools and the skills that you let it call along the way.”

“Our worldview is math reasoning is a true reasoning layer of AGI.”

“making small models in voice is much more difficult than making large models in voice. So keeping the quality while reducing the size of the model, that's where the big challenge is.”

“One drawback of speech-to-speech models is that since everything is integrated, when you go from a text model to the speech-to-speech model, you need to fine-tune it on speech data. So now it's the cost to switch the underlying text model is extremely high because you will need to re-finetune everything from scratch.”

“But now the workload is so large that there is room for specialization that will give you 10 X increases in certain domains, right?”

“if you look at coding as an application and you like look at these coding companies and how much they're paying for prefill versus decode, actually majority of their cost is prefill tokens, not decode tokens, because the context is just so large and it's switching all the time, even in agent modes.”

“If you have a business problem, you are maybe manufacturing something, maybe you can start with a generalist model, but then, uh, once you know exactly what the task is and you want to hone in on it, maybe it makes sense to replace that expensive thing by something like that, that is cheaper, you know, like a module…”

“It rewards the style more, more than the correctness because there is no correctness check.”

“And similar to when computers were introduced, productivity increased. Not initially, productivity actually went down. And you need to do diffusion in the economy to pick it up again. We might see something like that with AGI more broadly, and starts with software engineering scale showing up, uh, pretty significantly.”

“The models that we see today that we can play with today have been pre-trained on clusters that were built out, uh, a year or two ago. Um, because, you know, you, you need enough time to get the cluster running. You need enough time to do the large pre-training run. And then you need enough time to really post train…”

“An agent doesn't know that. And if you specify that to an agent, you could also just create the calendar invite, the meeting pipeline, and it, it doesn't increase productivity by much.”

“In my experience, there's maybe one or two of those things that make a larger difference than other things, but it's really a combination of many, many changes and many, many things from, from a very large team that actually makes Gemini three so much better than the previous generations of Gemini.”

“We're not really building a model anymore. I think we're really building a system at this point. Um, people have sometimes this view that we're just training a neural network architecture and that's it. But it's, it's really the entire system around the network as well that, that we're building collectively.”

“with the new paradigm of reasoning, you can get much more gains for the same amount of money because it's on this like lower and like, there are just discoveries to be made and these discoveries unlock insane capabilities.”

“So currently, and current for at least the Most basic ways we use it currently, it needs to be fairly verifiable. So there is an, is your answer correct or not? You prepare data for that. You can do that in mathematics, coding very well. Uh, you can do this in science to some extent, right? You can have test questions,…”

“If you learn to think for math, you can, you will sometimes do some, you know, some strategies are the transfer very much like look up on the web and see what they say and use that information. So some of these things are very generic and they start to transfer.”

“I think in, in general, the, the tech Labs are more similar to each other than people think. There are some differences, but, but I think if, if I look at it from the world, you know, from the university in France, the difference between this university and any of the tech labs is much larger than, than between one lab…”

“pre-training has always worked. And, and the beautiful thing is it even stacks with RL. So if you run this thinking RL process on top of a better model, it works even better. Than if you run it on top of a, of a smaller model.”

“So the understanding of what the models are doing on a higher level has progressed a lot, but then it's still an understanding of what smaller models do, not the biggest ones. But it's not so much that these patterns don't apply to bigger models. They do. It's just the bigger models just do so many things at the same…”

“these reward models tend to have a lot of problems and you can over optimize them much more easily because the reward models will pick up on features that are maybe emojis or something like this that you don't actually care about where RLVR is much better matched to like performance characteristics rather than style.”

“if the scaffolding is what really moves a lot of like from, from, you know, broad capability model to like something that actually has meaningful impact, that scaffolding is not just like, oh, only the labs of people are trained models can do it. Like the, the number of people can contribute to that. Both in terms of…”

“to get to highly capable intelligence that can reason, that can do planning, that can understand the world. You're almost better off putting a set of blinders and focusing on, on one proxy for that.”

“And what I found to be one of the best advice I got a very long time ago, whenever you find yourself in that situation, well, this is when you want to start finding the experts to join you, but how do you find the experts? Because you yourself aren't. And here's when you just start interviewing hundred plus people, For…”

“It is not that I can linearly add more GPUs and train increasingly a more larger model. If I do so, the time it takes becomes exponentially longer.”

“as soon as you get to a point where there's some convergence on an architecture that's looks like it's stable and is revenue generating and developers are coming to, uh, sort of work on it and confirm that it is like the thing, then you can flip towards doing a custom chip that's built to extract the most value out of…”

“So I think we're just getting smarter about how to do pre-training rather than shoving everything we have into a bucket and like seeing what happens. And so as a result of that, you might not necessarily have to spend the exact same amount of money to get a capable system.”

“The reason I think why task length specifically is interesting is because that's What allows you to delegate more and more work to language models, to agents. Now, even if you have a very clever model, but if it needs feedback or the interaction with you very often, then it really limits what you can delegate to it.”

“one direction of scaling RL and making it more stable is by improving this by, for example, putting more reasoning into your language model to generate much more high quality training data. That can then give us training that is much more stable, and then we can scale up much more easily.”

“I think nowadays, with the capabilities of, like, you know, top, probably cloud models, top OpenAI, GPT models, You don't need to do any fine tuning. You can take the model as is, ride your own tools, your own harness, and benefit from that agentic training. Because doing good agentic fine tuning is actually very hard.…”

“I think it's much more like a political, social problem of, like, figuring out how do we actually benefit from all these improvements, and, like, you know, bring the increases in wealth and productivity to everybody, and it's much less a technological problem. Which also means that we can't really solve it with…”

“top down, uh, structuring of research doesn't work in research organizations. I really don't believe in it because like you are not kind of hiring some of the smartest people in the world and open air has incredibly, incredibly smart people. To kind of tell them what to do, they need to, like, they need to figure out…”

“I don't think like you can just tell them all like, A few show with a few good things to do, and it will do them all needs to deeply understand its action and consequences to really, to really be able to choose the right thing.”

“And it's, uh, I think it's a never ending pursuit because like, even, even for humans, it's not super easy to, uh, to define what's, what, what, what, what do we consider a light? And I think as our, as our civilization will evolve, it will, the notion of alignment and, and, and the goals of humanity will, Keep…”

“generations of people have developed clever methods of encoding priors about how they think, uh, an artificial intelligence should reason. And encoding it into the model, and all of this gets wiped out by scale and, uh, and, and, you know, planning, like, like, basically, like, search and, uh, and learning.”

“a really high ROI use on, on your time would probably be to just go and look at the data and think hard about what, like the model is learning or doing and, and like make some tweaks that makes it like, there's so many, like you could even just The simplest things in the world will still deliver massive gains.”

“coding is a uniquely tractable problem in some respects for the, uh, The, the techniques that we have in terms, the data exists in many ways. You can containerize and run things in parallel. You can run unit tests, and so you can verify.”

“one of the craziest things about RL on language models, um, in the, in, like, the RL from verified rewards regime, is it's almost the simplest possible thing. It's, like, almost too simple to work. And this is, again, comes back to that question of taste, where Really, like, I think a lot of people thought this was…”

“the fundamentals of what coding is today, you are not able to use cursor. Um, if, if you don't understand coding”

“they will take time then to build the practice, they will take time to start feeding you with leads, so at the beginning, do not wait for them also to think they're going to source leads for you. At the beginning, you are going to be the only one to source leads for them, and it's really when they start understanding…”

“If you're just a product team, I think you're going to get leapfrogged by research. If you're just a research team, then, like, it doesn't have any real impact in the world.”

“it's less about one single, like, algorithmic innovation that's gonna change the entire field and more about how do you make sure that, like, those pieces are set correctly, um, because I think ultimately it's, it's a complex puzzle that has many different parts and you just need to know which ones are working and…”

“It used to be that you need really strong technical chops just to build an MVP to understand if, if you, this is something people wanted or not. And I think the reality now is that, um, that isn't the case. You can, you can usually figure out MVP or like, is there, there, there may be even early product market fit…”

“I think today there are so many products and companies coming out and vying for your attention that Launching something more polished so that when people use it, they're, they're wowed by it, by it is, is a way to stand out.”

“I think the failure mode is that we optimize for today's world. So we optimize for today's product and today's world and today's needs. And it's easy when you just talk to users and you get these requirements or you talk to enterprises. You just kind of, it's easy to make this assumption that's like, oh yeah, I'll come…”

“What we found is that a lot of the most interesting queries that people have, um, like would completely fail with that type of method. So for example, query might be like, what are all the things, um, I didn't do a good job explaining. Um, or, or tell me what are all the bugs that this user encountered, uh, that, you…”

“If you give like incorrect diarization, To a model, it'll oftentimes confuse it more than if it just has to try to infer who's speaking.”

“if you build a, uh, agentic system in the, in this context, Um, then the more general agentic system will generally outperform the more specific one in the longterm.”

“...my advice to companies building is definitely build for what the model will be able to do six months from now, not for what the model can do today.”

“I think for people that are learning coding today, it's actually harder than it was, uh, when I was learning coding, because not only do you have to know coding, because you still have to understand the languages. You still have to understand the frameworks. You still have to understand system design and all this…”

“superintelligence in these contexts of the large lab context is actually just being used synonymously with what AGI used to be used for.”

“You need taste. You need to have a vision of what you want, and you need to be able to articulate it. You know, those are like the fundamental, like literacy of a, of a person. A successful individual will be able to discern what they want and what they don't want.”

“And there's another emergent principle of you have to design from the model forward.”

“I think prompting is the biggest proxy for how would I measure someone's curiosity.”

“Critical thinking is something you can't really teach. It's something that you need to build around a domain of knowledge, and the only way you get a domain of knowledge is to, to work in that domain.”

“if you look at the individual repository or set of repositories, that code base isn't actually big enough to have a meaningfully fine tuned model.”

“They built so much infrastructure specialized to the transformer. And so it's like we dug ourselves into this. Well, um, like we now have chips that are being optimized explicitly to that architecture. And so to move architecture, it requires so much effort, energy, lift to rewrite everything and start from scratch.…”

“I think one thing that's different about agents and AI compared to other SaaS is that usually you're trying to do something that a human is doing in the organization. And To do that, you need the con, you need the same context that human has.”

“But if you go to it and say, give me a, a 40 page report on something I don't know much about, you can't trust any line of that report, because most of it will be right, probably, or it will be roughly right, but if there's anything, but you won't be able to depend on any statement in that report actually being…”

“I think that stickiness, I don't think it's a network effect.”

“And there's a trap with these humanoid robots, which is some people look at them and think it's AGI, and it's not, it's just a robot that's got legs instead of wheels, but it's still a robot.”

“Silicon Valley really has this problem in not understanding that other industries are hard. Like, the airline business is hard. They're not just idiots. It's difficult.”