The Wisdom Wall

“Product managers are effectively glorified UI managers, so we're trying to make sure that us common folks can interact with data and some sort of engineering algorithm behind it because we're not smart enough to talk to the engineering algorithm ourselves.”

“If you're an advanced math problem, and you're doing seven times eight, and seven times eight to you is still a calculation. Right? You're doomed, right? There's just, there's no way around it. You'll start making careless errors.”

“If you don't give them a fact base, then their opinions are just making stuff up. You have to reason over a fact base.”

“I mean, I, I think there's a lot of truth to that except, um, you don't want to go too close to that stuff because, because you'll actually kill the propulsion system in analyzing it.”

“their ROIC is, like, extremely high on CPU and storage, and, and to assume that it can, like, translate over to GPUs is, is a bit of a fallacy, which is why, which is why a lot of these companies are moving in, right?”

“Our thought was that this was the time where you can actually start a, um, you know, a generational frontier lab that does not need to be coupled to a, you know, to a big cloud provider, uh, because if you do it right, you'll actually be able to generate, um, you know, sufficient revenues to not have to be acquired or…”

“There's no such thing as generalization. There's just bringing the test distribution into train.”

“there's this sort of like idea in Silicon Valley that you should learn a lot from failures. And I, I, I'm not sure that I agree a lot with, I think that actually people probably learn a lot more from their successes and, you know, companies fail for like, Many dumb reasons. And like, it's really hard to sort of take a…”

“If you do it voluntarily, you just make yourself less powerful and you let the worst actors get ahead of you. Um, you could say, well, we'll try and try to sign a treaty. Um, we will not assume that the treaty will be followed. Like that would be very imprudent. You would actually need some sort of threat of force or…”

“Safety isn't, as I've been I'm trying to reinforce not really that much of a technical problem. This is more of a complex, um, geopolitical problem with, with technical aspects.”

“one thing that has been a Has been, has been slowing the progress in bio is the fact that we have always been super hypothesis driven, and I think it has, the reason is that a lot of these experiments are expensive, you know, that they take a lot of time, a lot of resources, but I think now is really the time that the…”

“They will leave their academic program because they want to commercialize their research, which to me is like a huge red flag, because that means you are, you're saying, I'm not going to be flexible on how I solve the problem. I'm going to force my solution into the, you know, square peg into the round hole. And I…”

“I think I want to challenge like Paul Graham says, do things that don't scale. I think that that's game over for that statement, right? I think now we only have to do things that scale.”

“And so the SA was a way to use Amazon as a way to tell a broader story about the history of our antitrust laws and how we had evolved from, uh, you know, Viewing this issue of, of dominance and, and monopolization through a broader prism, uh, and a broader understanding of the different ways that companies can exercise…”

“I think it's incredibly dangerous to try and make predictions about future capabilities, about where the technology will go and make rules, legislations, laws. Like prohibiting or enforcing or requiring certain research directions, um, through a theoretical lens. It just like hasn't empirically worked.”

“Assuming that the model should be well behaved is, I think, a wrong assumption. You need to make the assumption that the model should know everything. And then on top of that, have some modules that moderate and guardrail the model.”

“You don't set up a flare and say, everyone should pause for six months. Oh, look, eight billion people paused. On your theory of the universe, the UN would be a highly functioning organization that we would all use for a whole bunch of things. It doesn't work.”

“And you just can't have AI built by UN committee. It just doesn't exist. And it's one of the things that academics mostly don't understand because they've never, most of them, not all, of course, but most have never built anything, don't really understand how these organization works, don't understand how technology…”

“culture shorts and sifts. You attract the right ones and you start losing the wrong ones. So it's actually quite perfect. If people are leaving, they're just not your DNA. They're not your, your blood type.”

“the people who use the LLMs for building stuff, Understand it better than the people who actually did gradient descent and train these models.”

“This is why I'm interested in the reasoning direction, because I think there's this whole other dimension. That people are not scaling right now, which is the amount of compute at, uh, at, at inference time.”

“Now it seems that you're coming to a large legacy enterprise. They really want to adopt the ad because they have to, the problem is they don't know how to do it. They understand that their processes are very long. There's something that takes a year or, or, or two or more, but they need to have it now. And the only way…”

“If you can, if you get vision, hearing, balance, and a kilobit per second of motor control, you're halfway to the matrix. And this takes you into some like really trippy interpret reinterpretations of medicine.”

“if you're going to reboot nuclear and you're actually believe in, in the atomic energy century, which I absolutely do, it's probably not going to look like what it did in the sixties. It's not going to look like civil infrastructure. Because we're not that good at that anymore. And what we're way better at is…”

“What was missing is that people did not realize that you're going to have to pursue that model through hardware iteration. And that's where really where Valor is unique is that Yes, we believe in SMRs. We believe in reactors that are manufactured. But we also know that in order to get there, you have to do it through…”

“And people, um, Are, at least in my experience, uh, collectively, at least investing against the fact that they believe the speed is there, which is different from the market size. People are conflating the two things right now, in my opinion.”

“Yeah, I mean, the physical compute basically, um, reinforces an oligopoly market because what it does is it creates a ceiling On the rate of progress any single lab can get, if effectively you assume the compute is roughly pro rata across the ecosystem to the big labs.”

“if you're trying to solve that three to five mile delivery, In dense suburbs, which is where most of the deliveries happen, the right metaphor is probably a autonomous motorcycle or a scooter or a bike profile vehicle. And, uh, you know, it doesn't even be 4000 pounds. It's probably, you know, 300 pounds, uh, but it…”

“It's like, okay, if you like dumb down the problem, maybe the models do well, but like for some reason, and when we actually have it, With the enterprise data and all the real stuff, it's not performing as well.”

“And so I think the proper way to, and so my claim is the proper way to evaluate the models now is you either have some kind of budget for the benchmark, whether it's tokens or cost or time or whatever, or you plot the performance as a function of the amount of test time compute that's going into the model.”

“it's really easy to show you can do much better than previous benchmarks or, or Uh, previous, previous models on benchmarks by just, for example, scaffolding a bunch of models together. Um, so if you say, okay, well, we're going to, instead of just running this model once, we're going to run it five times and take the…”

“The preparedness frameworks and responsible scaling policies, they don't really account for the amount of tests I'm computed. They just say, okay, well, what's the capability of the model? The problem is we're in a world now where the capability of the model is a function of how much money you put into it, basically.”

“the model release cycle is every, every couple months we put out a new model that's even more powerful, and so the cost of disproving the Erdos unit distance gesture drops by, like, 10 or a hundred x with every model release cycle. Probably, in some cases, more.”

“I would talk to researchers about, um, we, it makes sense to show the benchmarks with an x-axis, whether it's tokens or cost or time, there should be an x-axis, and everybody would say, like, yeah, that makes sense, we should do that, but. Well, really, their response is people expect us to To publish the grid. And…”

“The next thing is look at, it's very important from day one, you'd have to target the first customer. And usually I like the customer is hyperscale. They have the scale. If they like what you have, they're willing to pay millions of dollars next few years. And even giving some warrants is worth it because you have a…”

“If you're building tools that are this complicated, you kind of want to have a 10 to 15 year time horizon on, on building out these efforts. And then the scale of capital required. I mean, I guess there's no rule that said that you couldn't do it as like an incredibly well-funded startup, but I think that this just…”

“in order to make progress in AI, you don't need like many, many hundreds of AI researchers, um, or thousands or anything like that. I think you can really make progress with, um, you know, a very strong group of a dozen or a couple dozen people.”

“Most importantly, you'll have private evals because we know all the evals out there are good, interesting, But they're not really that critical at this point because they're all can be maxed. And so the point is each company will have its own private eval.”

“that idea that you can build a platform layer that someone else can then extend out, um, and build their own intelligence layer in this case, I think is everything right without it. Why have a developer conference? I can just come and have you all sort of just worship at the altar of one model, but that's not a…”

“most people love outcomes until they have an outcome, because once you have an outcome, it's like giving away royalty, right? I mean, I've talked to customers who love, you know, outcome based pricing, and I say, I'm all in until they, oh my God, like, what are you talking about? You're sharing in my outcome? No, no,…”

“what we've done with AI is when you make your existing teams 30, 40% more efficient and they can handle more customers, um, it changes the whole mindset of the organization. Now you're growing, you look like a software company now where you're now growing with High incremental margins. And that allows you to invest…”

“to the extent that that is encoded, um, in a model, I think a lot of their business will, um, be at risk, but to the, to the extent that it is encoded in workflows, um, that is where they will be able to develop mode.”

“hey, go find, go prove to yourself with the best in class model that you have something worth optimizing. Um, and and I think, you know, A lot of, you know, if a customer comes to us, was that meme, which was like, it was like two years ago, it feels like there's no GPUs pre-product market fit. It's like no…”

“I think one fallacy is thinking about intelligence as a scalar. We've consistently seen these systems have a very odd spikiness, and it's actually possible to architect a system that is world-class on some math domain, but then you could do some perturbations to the questions and actually degrade it substantially, so…”

“even if they don't work, I think to a large extent, you feel like it's a skill issue. It's not that the capability is not there. It's that you just haven't found a way to string it together of what's available. Like I just don't, I didn't give good enough instructions in the agent's MD file or whatever it may be. I…”

“to get the most out of the tools that have become available now, you have to remove yourself as the, as the bottleneck. You can't be there to prompt the next thing. You're, you need to take yourself outside Um, you have to arrange things such that they're completely autonomous and the more, you know, how can you…”

“A research organization is a set of markdown files that describe all the roles and how the whole thing connects.”

“So if the barrier comes down, then actually you have the Jevons paradox, which is, like, you know, you actually, the demand for software actually goes up. It's cheaper and there's more”

“even with other research, like, OpenAI or, or, you know, uh, Anthropic or these other labs, like, they're employing, what, like, a thousand something researchers, right? These researchers are basically, like, glorified auto, like, you know. They're, like, automating themselves away, like, actively, and this is, like,…”

“Like, if you're inside one of the frontier labs, like, there are certain things that you can't say, uh, and conversely, there are certain things that the organization wants you to say, and, you know, they're not gonna twist your arm, but you feel the pressure of, like, what you should be saying, you know, because,…”

“And I think if you're outside of that frontier lab, uh, your, your judgment fundamentally will start to drift because you're not part of the, you know, what's coming down the line.”

“I think this is honestly a, a really key thing and something that a lot of companies get wrong is just like doing one thing and then just like, like sticking with it. You really do have to keenly aware of what the current state of the models and the technology is, and then designing the harness, the system and the…”

“the upshot is I think if you do it well, you can be much more ambitious about what you're building. And also make it much more robust than you could have done, uh, with, with, with humans writing it. And then the flip side is if you do it badly, it's all slop.”

“I think that fundamentally, um, you know, there's, there's so much demand for software and there's so little supply of engineering and reality relative to that demand that as you add this enormous boost of productivity to software engineers, um, it just gets soaked up. Right. Because there's so much more stuff to build…”

“one thing I've seen is that if you have, uh, An engineering identity that's based on that, like a value based ranking of difficulty or skill, like the, the specific types of engineering that are considered, you know, impressive or high status can actually be like less hard for agents.”

“if you can generate an enormous amount of code and no one is reading it, you don't know the quality of the code. Nobody deeply understands the code base and there's more fragility, right? It's like the slop problem, but instead of it being like vibe coding slop for random websites for non-technical people, it's vibe…”

“Because often for companies, there's about a 12 month window where your company's the most valuable will ever be, and then it crashes out. For a very small handful of companies, the answer is you should never, ever, ever sell. For most companies, the answer is you should sell when the timing is right.”

“The best way to defend against this is to build a bundle.”

“And so the, the difference with AI, Is the velocity of change is so high that what normally would have taken a decade, and you'd have a normal decade-long displacement cycle, is now happening in a year or two. And that's really the, the reason that these things are so turbulent.”

“So I think my view is the EV adoption in the United States is a reflection of the lack of choice. Uh, there's one set of really great choices with Model F and Model Y. I think there needs to be many more.”

“So yes, wipe coding will allow you to create an app fast. You have a great use case. You have some disruption in mind. That's excellent. But then there is a lot of things that you need from a go-to-market perspective to be able to break in, pass all their checks, governance, security audits, and things like that.”

“Because you will have my attention as a leader, if you say I'm going to replace it, it's going to be cheaper, it's going to be faster, it's going to be better, and oh, by the way, I have disrupted pricing, and for the value you get, that's when you pay me the money. That's a conversation I'll have every single day.”

“the main part that I think is different in audio space is that you don't need the scale as much as you need the, Architectural breakthroughs, the model breakthroughs to really, to really make a dent.”

“the thing that will really give that long-term value is the ecosystem that you create around, whether that's the run and distribution, whether that's the collection of voices you can have, the collection of integrations you can build, the workflows that you can build. Um, like, I think that's, that's the way we can…”

“People get very focused on clinical trials, um, because they're like the biggest, the biggest line item. And they're important, don't get me wrong, but I think it's actually a bit of a red, red herring. Where, ah, yes, there are operational problems. Like, some studies are designed badly. It's hard to recruit patients.…”

“No, look, I think the most vulnerable people are where there is poor loyalty to the UI.”

“We never believe that you can take three or four and spit and shine and make it look like one or two, because one or two don't go away. They trade as one or two for a reason.”

“Nobody has two sales forces deployed in an enterprise. Nobody has two workdays deployed in an enterprise. Nobody has two SAPs deployed in an enterprise. Why? Because you need end to end visibility, a singular workflow, a singular set of analytics to solve the problem... So if cybersecurity has to survive in the long…”

“time back by far is the biggest motivator of kids where if you took our Westlake high school, our, you know, in Austin, where they do six hours a day in class, four hours a day homework, if you're on API track, Right. You can't pay those kids enough money to be active. Right. They are just grinding it. And if they love…”

“I don't think if you build it, they will come is true. And nor would I say that, you know, Apple or Steve Jobs is a story if, that if you build it, they will come. To me, you know, Apple or Steve Jobs is a story that, um, if you have extraordinary will to power And you see reality as malleable, and, uh, you believe, as…”

“The solution to help put the genie back in the bottle is, I think actually the DoD's deal with MP Materials offers a good template, which is you need an anchor buyer and an end customer, and you need a price floor. So you need to agree with the supplier, in this case MP Materials, on a floor for a price, because what…”

“It's like a deeply intellectual exercise, and I think the term vibe coding makes people think it's easier than it is. So frankly, after a day of using AI-assisted coding, like, I'm exhausted mentally, right? So I think of it as rapid engineering, where AI is letting us build serious systems, build products much faster…”

“Founders that are on top of Gen AI technology, thus, you know, uh, tech oriented product leaders, I think are much more likely to succeed than someone that maybe is more business oriented, more business savvy, but it's not, doesn't have a good feel for where AI is going. I think unless you have a good feel, For what…”

“The best engineers I work with today are not fresh color strats. They're people with, you know, 10, 15 or more years of experience, but they're also really on top of AI tools, and that, those engineers are just completing a class of their own.”

“their very algorithms, as they, as they come across a piece of content, they prune off that content Which is already kind of part of the meaty part of the cheese, whereas the parts that are in holes are actually super valuable to them, and so I actually think that if we could create a market where you're rewarding…”

“And one of the measures I've been thinking about is let's say the world inferences, I don't know, maybe it's called 10 quadrillion tokens a month. We don't know what the number is, right? What share of those tokens are being inferenced on American hardware on American models, right? And how do we maximize that market…”

“Interpretability is still a nascent field and you could very easily see ways where You plug a model into a cursor or windsurf and you generate a piece of code. And then two years down the road, it turns out that code had a little if statement saying, if I'm running in some piece of critical infrastructure, go do…”

“the important part, I think, is to be able to tweak every part of the system from, you know, the product features to the agent design to the model training, uh, in order to build the best overall system. And if you are capped in which parts you can change, like if you can only change the products and agent design, then…”

“we're in this brief period in history right now where, um, the RL flops are still manageable. Like you can, you can, you can really have a best in class, um, product if you're focused. And yes, you'll need to put, you know, you still need a decent amount of GPUs, but from a, but from a flops perspective, it's nowhere…”

“the reward problem in itself is at the time I called, I thought it was AGI complete. Now I'd say it's ASI complete, but by the time you have a neural network that can accurately verify any outcome, that is probably a super intelligence.”

“when we're generating synthetic data, um, there is the only scalable path is really reinforcement learning.”

“The reason code is special is if you believe that, uh, the way a language model will interact with almost any piece of software is through function calls and therefore code, then if you build very capable reasoners, um, coding reasoners that, you know, are sort of purpose built for organizations. So you've solved the…”

“these artisanal software companies really can't afford to invest in a set of underlying capabilities that are ultimately what make these applications powerful for their customers. Um, and that end up being repeated across a lot of these different application areas. So things like permissions and reports and analytics…”

“at Zenefits, we kind of had this, this theory that was ultimately, I think part of like the downfall of the company that, you know, we could move through the market more quickly by doing a lot of things manually. And then, you know, we would just replace it with software over time.”

“our view was there's no path to autonomy. There's no path to like this next generation because without the manufacturers in the loop”

“there's a lot of focus on embeddings and vector search over the last few years, but that's actually only one part of, ah, building a good search system. Because if you think about an enterprise, ah, imagine a company that has been around for a few decades. You know, they have tons and tons of information spread across…”

“we started to hear from businesses that, oh, I'm scared of good search. I don't want a good search product in my company because I have all these governance gaps. I have like, you know, sensitive information all over the place.”

“People talk about hallucinations as a big problem with AI models. You know, we feel like, you know, a bigger problem for us is not even hallucinations. It's about Like, you know, most of the times you can't even, you know, find that information. Sometimes it's not there. People are asking questions, but nobody wrote it…”

“We've in the past for our reasoning models talked a lot about test time scaling, and I think for a lot of problems, uh, you know, without tools, test time scaling might occasionally work and, but at some point the model is just kind of ranting in its, in its internal chain of thought. And, uh, especially for like some…”

“And so I think we're going to see an immense amount of eval creation for, like, agents, uh, and that is the largest barrier to Automating most knowledge work in the economy.”

“And we found that actually that mounting RGB cameras to the wrists ends up being very, very helpful and probably giving you a lot of the same information that tactile sensors can give you.”

“We have the machine language layer, you know, which is Python or Ruby or Rust, right? Those are effectively abstractions of the, of the chipset, um, and the machine instruction set. Um, and that's the last layer that's deterministic, right? Like programming language inherently when I, it does exactly what I wanted to…”

“I think we now have the tools to be able to, in a very Kind of molecular way, microstage, what I call our biostage disease on a trajectory that isn't four stages, but it's 75,000 stages if you want. And the key for that is that you can then look at what mechanisms are turned on and off in any one disease during that…”

“People criticize these models for hallucinating, but if you think about it, the process of scientific research just involves hallucination, right? That's what creativity is.”

“You basically have no product until it achieves superhuman safety performance. Whatever you establish that is like just slightly better than humans or 10 times better. I think most driverless cars that are on the road today fall somewhere in that category. And to get there, uh, it means there's no MVP. There's no…”

“too much anthropomorphism can, I see I screwed up there, can it can imply like a set of human-like behaviors that don't exist in that product. And I think Rodney Brooks wrote an essay about this basically saying with robots in particular, the appearance of them sets the expectation for what that product will do. And so…”

“If you're a local burn boss or fire chief and you want to do a prescribed fire, You know, it is the right thing to do, but it's kind of risky. If it, if it escapes, if it burns someone's property, if you start the next, you know, Palisades fire, you know, you're gonna lose your job. This sort of risk that's where the…”

“The water issues were related to there's three, one million gallon tanks, um, as houses burned down, the plumbing opens. If you've got like a five eighth or half inch plumbing connection into a house, that's going to run at 20 or 30 gallons a minute. You have 500 of those going, you've got really significant water flow…”

“If, you know, PG&E spends five billion dollars undergrounding lines or upgrading transmission lines, They get to mark that up by 15% and pass that through to ratepayers. They love it. That's how they make money.”

“the hard problem, and it's kind of like that, the early part of the curve has been getting like, you know, even steeper and steeper, but that's not where the complexity is. The complexity is in the long tail of the many, many, many nines. And you don't see that if you go, you know, uh, for a prototype, if you go for,…”

“It's not going to be, you know, the system prompt kind of a, you know, for those who have done web development, it's kind of like view sourcing on a webpage. Like you can go to google.com, you can view source. No one so far has kind of done that and built another trillion dollar company that took out Google by doing…”

“we're starting to enter into a sort of flat part of the curve, um, and we're certainly past the point where if you just interact with a model, You can know how smart it is. Like the, the vibe checks, they're losing utility.”

“I don't need to go double the size of my supercomputer to hit a requisite intelligence threshold. I can just double the amount of inference time compute that my customers pay for.”

“You know, I say that the new unit of computing is the data center.”

“Like, grain prices, or hurricanes, or Brexit happening or not, or election, Trump versus Kamala winning, that's a risk that exists already, right? Like, it's not like you and I are creating this risk so that we can bet on it. This exists already, and some people, like, if you run a climate tech company, you're…”

“So how do casinos actually advertise their products? They, they, like, show you the one person that made 300,000 dollars in the slot machine. People get excited and everyone goes and does it. And, you know, everybody knows the odds, et cetera. YC kind of does the same thing if you think about it. Like, you know, we,…”

“You might plateau in the right place for a driver assist system. You will plateau at the wrong place for a fully autonomous system.”

“So I think until open source can really move the needle on one of those two axes, it's, it's going to be tough, uh, for it to be adopted broadly.”

“And I think the, the, the big insight or the crazy, you know, non-intuitive thing about, um, LLMs is that something trained on the internet outperforms, uh, what an enterprise can produce with their own data trained, um, on data in a data warehouse.”

“Probably the most consistent thing I've seen is companies kind of walking back from the, uh, illusion that totally free form agents will solve all of their problems. So I think maybe like two or three months ago, Many of the pioneering companies went way down the agent rabbit hole. Um, and they kind of realized like,…”

“the biggest thing I've seen over the past six months is, People dropping the use of frameworks.”

“What we've seen instead over the last couple of decades is that digital markets are different. And in fact, they can lend themselves to tipping. Uh, it can actually become much more difficult to break in Once you have a firm that's become dominant and is protecting that dominance through network effects, through…”

“if you're selling to the company that's already the monopolist in a particular market, that is more likely to raise issues than if it's, you know, selling to a company that doesn't really have a presence in the market.”

“what we found in practice is that broad category of technology investment is woefully Uh, insufficient for almost any meaningful customer experience. If you think about, you know, all of the interactions you've had with brands that you care about, what percentage of those conversations were asking questions? Probably…”

“If every time there's a new release of an AI model, somehow it decreases your value, it probably indicates you're, you're not actually a solution. You're, you might be a slight value add on top of the models. I think there's a number of startups that unfortunately sort of smell like that. You know, it's not, not…”

“when people talk about the scaling loss in neural networks, the scaling laws are actually a, um, uh, to a large extent of a property of the transformer. Before the transformer, people were playing with LSTMs and stacking them, etc. You don't actually get like clean scaling loss, and this thing doesn't actually train…”

“I don't think that the neural network architecture is like holding us back fundamentally anymore. It's like not the bottleneck, whereas I think in the previous, before Transformer, it was a bottleneck, but now it's not the bottleneck. So now we're talking a lot more about what is the loss function? What is the data…”

“currently at the current AI capability, I don't think the models are good enough to create a good course. Uh, but I think they're good to become the front end to the student and, uh, interpret the course to them.”

“And the correct answer is mostly like, um, I would say like math, physics, CS kind of disciplines. And the reason I say that is because I think it helps, um, uh, for just thinking skills. It's just like the best thinking skill core. Uh, is, is my opinion.”

“If you, uh, your end goal is to have a system that can do everything, uh, you can reduce that to building a system that can build that system. And so that minimal system is a system that writes code and comes up with ideas and can validate those, uh, by writing code and running experiments”

“Retrieval selects a subset of data for one completion. Our model sees all the data all the time. Clearly, a subset of data for the whole completion is a subset of all the data all the time. So if retrieval was optimal, our system could learn it. Um, and, uh, it just turns out that it's not optimal. That's the sort of…”

“the only way to, to, to sort of reasonably approach this is to iteratively ask your model, uh, to solve, uh, uh, alignment and safety at that stage, not, not, you know, surely you can also ask it to solve your product level problems, but like that, that's, that's nice, but that's not, that's not the fundamental…”

“The reality is I really truly believe that capitalism and competition are the only chance we have to provide a, an optimizer that is capable of getting us to the right place. I think we need the right guardrails to do that.”

“I think people are getting more sophisticated at thinking about cost in a more of a life cycle way, and that's actually leading to the workload shifting from large pre-training more and more towards things like batch inference. Which is actually a really, really horizontally scalable workflow that you can paralyze. You…”

“the writing the email is the easiest part. That's actually not that hard. The part that is the hardest is aggregating all the data and essentially building this, um, this database of your whole team. Like who is every account in the market that could buy Brex potentially. And what do we know about them? And how do we…”

“The contrarian view would be, I'm not sure it matters that, that we achieved AGI. I think it might not look like Hey, like, it's gonna be exactly like the cognitive task we can do, and then we reach parity. Um, it's gonna be a distribution of things that these models can do or can't do, and that to me feels like what's…”

“I think that, that, you know, you'll have some use cases addressed through solution companies that can be very big, um, you know, uh, each individually. And yet, you know, I, I still think even if you added all of them up, they're still tapping into such a small fraction of, you know, all of the actual departmental…”

“in the design context, um, one thing that really matters a lot is the iterative loop and being able to keep going back and forth, uh, to an agent and give more instructions over time. If you just kind of like go to first principles here, there's so much that you're not able to communicate via a prompt.”

“And so actually to really get, like, perfect, um, even just TTS or, like, speech-to-speech, um, you, you actually really need to have, like, a model that has, More understanding, like at least of the language, but kind of like, it's not really an isolated component anymore. And so you have to start getting into these…”

“The key to the scaling of these large language models and the, you know, these, these, uh, uh, language models in general is the ability to scale data. And I think that one of the fundamental bottlenecks to, you know, what's, what's in the way of us getting from GPT-IV to GPT-X is, you know, data abundance.”

“The model shouldn't know about music theory. Um, you don't tell GPT, this is a noun and this is a verb. GPT figures it out. Um, if I tell my model, there are only 12 tones. My model will only know how to output 12 tones. If I tell my model there's 50 different instruments, I will never get that unique sound. And so,…”

“And, and if you are meta and you have Somewhere between, you know, 22,000 GPU clusters and 350,000 GPUs, um, available, uh, then continuing to train past, like, supposedly optimal points, like, does improve performance apparently, um, and doesn't just fully asymptote as soon as maybe a lot of the research community…”

“what works really well as you increase scale is just predict data. And that's what we do with text. We just predict text. And that's exactly what we're doing with visual data with Sora, which is we're not making some complicated trying to figure out some new thing to optimize. We're saying, hey, the best way to learn…”

“I think there are all these, there are like lots of tricks that sometimes get you like 10, 20, sometimes two X improvements. But I think the number one trick is like really just like that last phase of You know, a supervised fine tune where you're finding like really great curated data.”

“Whereas like with vision, at least you can like, you can like make a robot that just like travels down the street and like just keeps taking pictures of everything. Um, you can get like infinite training data, um, uh, with vision, but it might be trickier to like sort of filter and clean internet data, especially as…”

“You want to build like the equivalent of like the keyboard and mouse to the internet in the physical world. That equivalent is a human form. You can do everything a human can. And the world was optimized specifically for us.”

“That, that actually really doesn't reflect a lot of RAG use cases in, in my opinion, because like that's the needle in the haystack is like, okay, given this long context, can I find a single information point? But oftentimes RAG is about seeing multiple information points and then reasoning over them.”

“at the early stages, a lot of the data doesn't exist, right? Like you'd have to capture real world data. You have increasingly meetings over zoom, but, uh, you would want to capture a lot of information about people. Um, so much of it is access and the information about who is like leaving and like a hundred X engineer…”

“Interestingly enough, many of them actually start behaving quite similarly in terms of level of accuracy, even though without RAG, they, they actually have quite different behaviors. So it's sort of both like a uniform improvement and a little bit of leveling the playing field.”

“Foundational models get it fundamentally wrong. When we learn how to Build the subsystems of AI correctly, and for each one of them to do their roles optimally. Either we're going to do, be able to do the, to achieve the same tasks much cheaper, faster, better. Or we're going to still want to use the same amount of…”

“Fundamentally, SaaS, software is all, we're all in the same information, people, paper pushing activity, right? It's like a piece of paper coming in front of you. A human, like, changed a couple bits, pushed to another human. Language model can do some form of this now.”

“I think organization might be, we might be moving away from the organizational world. Why do you need to organize because you can retrieve? Why do you have index? Like index initially are file cabinets and little index are sitting on top of so you can find things quickly, right? And they're index based on certain names…”

“And the only way to collect enough data is to build fleets of robots that are actually creating value for customers that, so that you can collect those data in production. Because even if you try to scale up data collection in a lab environment, There's a limit on how much you can do that.”

“you kind of don't need that when you are in a more Restricted domain, like robotics, um, because like you actually could have so much data coverage that your test scenarios are just part of your training scenario. Um, so to some degree, like we actually don't need to rely on this strong form of scaling law to whole,…”

“when you think about simulation in self-driving car, like we are really mostly thinking about systems that hopefully don't physically interact with each other, right? Like if two cars get in contact with each other, that's a really terrible thing, right? And so the simulation there is more about simulation of…”

“For people that study reinforcement learning, we call it behavior cloning, which means you're just asking the AI to clone the behavior of another agent. And that is like one of the most primitive way possible to train this type of systems. Like, because if you're just mimicking something, like there's a natural ceiling…”

“I think if you talk to a lot of RAG practitioners, uh, you'll find that the kind of, like, dirty secret is that keyword search can probably get you, uh, more than 90% of the way there.”

“doing the simple thing, it, it establishes a baseline. Like oftentimes you'll find that like the, the, the doing the fancier thing is often sexier. Uh, and it's certainly these days it's like trendier, right? Cause you can kind of claim the mantle of like, ah, you know, I made a, I made my own LLM beyond LLM and it,…”

“I think you're still going to want to do RAG anyways. Like, even if you have fine tuned models in the mix, RAG is still sort of this, like, last mile data or, or context.”

“What I think we're starting to see right now is a, uh, the pendulum starting to swing back in the sense that there is a greater understanding that you really need a bit of both. You need that, uh, hugely powerful pattern recognition that we get from deep learning, but you also need the ability to reason about things…”

“the critics think they're virtuous because they go, oh, there's this, there's this danger that we should, we should trumpet, and you're like, well, actually, in fact, you may be doing more harm than you're doing good in your attempt to be virtuous, because by trumpeting the negativity, you're not shaping where we could…”

“one of the things with the cloud providers too, is because you have sort of default customer demand, you launch one of these services using the open source. In many cases, you'll, you'll have existing customers that want to use it. And so then you're immediately getting customer feedback, you know, to help improve and…”

“really what you're doing is you're taking services revenue, which is very people intensive and low margin, and you're converting it into higher margin software revenue, but less of it. So maybe you take that five trillion and you turn it into two trillion, But it's 80% margin two trillion versus 30% margin, right? The…”

“Like, we know even in theoretical senses, like, they cannot learn to do multiplication in the general sense because it literally doesn't fit in the context window, right? Like, multiplication, they can learn to do addition in a modular sense, and they can learn to do it actually almost perfectly if you train them in…”

“It's not like, oh, magical, you know, we train a giant model and stick everything into it and then magically it works. Like it does not work. Um, it'll get better at random parts of the agent loop, but that's not what we want.”

“The appetite for compute is truly endless as, as now, as, as now clearly seen, but we realized that we will need a lot. And a nonprofit was, wouldn't, wouldn't be the way to, to, to get there. Wouldn't be able to build a large cluster with a nonprofit.”

“Because if you imagine how an AGI should look like, it has to be some kind of a big engineering project that's using a lot of compute, right? Even if you don't know how to build it, what that should look like, you know that this is the ideal you want to strive towards. So you want to somehow move towards larger…”

“I would actually point out that the main thing that's lost when you switch to the smaller models is reliability. I would argue that at this point it is reliability that's the biggest bottleneck to these models being truly useful.”

“These are fairly well-known ideas in AI that the cortex of humans and animals are extremely uniform. And so that further supports the, yeah, like you just need one, you need big, uniform architecture. That's all you need.”

“I think that will happen when those systems become reliable in such a way as to be very autonomous. Right now, those systems are clearly not autonomous.”

“technology is already reproducing using the minds of people who copy ideas from previous generation of technology. So I claim that the reproduction is already there.”

“I feel like they're in general, people are viewing, um, AI as this continuum where it's like, it's a CNN, RNN, and now we have transformers and it's just a straight line. And instead, obviously it's a big discontinuity in terms of capabilities. And I think most people still don't think about it that way, or at least I…”

“there's a, you know, like a truism that all cybersecurity products are just, you know, complex insurance policies or whatever, right? But the, the, the point is like that, that mushiness exists. And so what has resulted in, in, in the free market here is these incredible distribution machines, right? You have, you…”

“a lot of the best VCs like never, you know, make bets in cybersecurity because, you know, at best you're going to get a two hundred million dollar takeout to Paladin networks or whatever, right? That's the typical outcome”

“In the, in the latter part of an industry, when things have matured and sort of saturated, that's when you do the hard stuff. But in the early days of a new market, you just want to do the easy stuff, because that's, that's very tractable, it's faster, it's easier, you know, higher velocity.”

“HR people are tricky to hire because in HR you've got recruiting people, you get compensation people, you've got culture people, you get DNI people, you get benefits people. And, you know, if you've got a person who runs HR, they likely grew up through one of these things and doesn't know much about the others. Very…”

“in order to 10 X or even a hundred X use cases and usages for AI outside of that, there's things that could just be done on existing models today. So you don't need to wait for GPT seven or whatever. You could start with GPT four or GPT 3.5 and add these things.”

“There's a, there's a, a great phrase that I love, which is copy and paste is always better than a bad abstraction. And a lot of the worst code bases are the ones that are over abstracted.”

“At the end of the day, in my mind, that's the one and only thing we know really works. If you want to push deep learning forward is to make it faster and more effective and more efficient on a given piece of Hardware.”

“And if you want to look at, say, the biggest differences, for example, between the transformer, as it was described in the attention is all you need paper, and some of its ancestors, like this decomposable attention model, the big difference is just that the transformer was implemented by folks like Norm and Diffuse,…”

“The other main contributor, I think, to the success of this architecture was optimism and hope. So suddenly you were in a situation where, for whatever reason, a bunch of things that people tried with this started to work, and then more started to work, and that's not coincidence. It's really just because ultimately…”

“And ironically, and this comes back to a question that many people ask, I think around, does it make any sense to train on generated data? Because information theory, family information theory, very clearly says, nope, you're not going to get more information out of it. You can do it all you want. But there is an…”

“There are people trying, and I think it's worth trying. I, I'm not super optimistic about that. I think it'll work for some cases, right, where it's simple enough that we can get it. I think there are many cases where it just isn't, right? Like, say, climate and weather forecasting. I just don't think we're gonna get…”

“情報と概念を指先で使いこなせれば、これらのツルハダレよりもあなたを加速させるでしょう。逆に、サンバイナーナーの計算をするために電卓を使ったり、第二次世界対戦外走行とかを調べるためにググルに行かなければいけない子供は、世界に取り残されてしまうでしょう。”

“So now every coder is not going to have an army of coding assistants. Every writer has, has a writing assistant. So people are gonna have to move, Into the managerial ranks quite quickly and figure out, okay, instead of being a coder, I'm now an architect. Instead of being a writer, I'm now an editor. And in order to…”

“The more that you have rapid access to knowledge in your head, the more that you are fluent with your mathematics, that you have information and, and concepts at the tip of your fingers, these tools will accelerate you more than anyone else. And the kid that has to get the calculator for three times seven, or has to go…”

“the practicalities of an enterprise is there is either an implicit or explicit, and often it's kind of fifty-fifty or some combination, implicit-explicit state machine that represents that enterprise. So the idea that you're just going to have an agent that kind of comes up with a plan and does things is it's not going…”

“this is what naturally happens to human beings. It's, it's just, it's innate. We slow down to a glacial pace unless there are People who are going to drive tempo and pace and intensity and urgency. That's what leaders need to do because people naturally slow down.”

“CEO jobs are insanely confrontational, which is not human nature. We don't like it. We aren't naturally confrontational. We avoid it.”

“in a world of AI, if you don't have highly organized, optimized, sanctioned, and trusted data, what do you want, you know, your models to do? Just kind of train on, on, on a data lake. I call it a landfill. You know, I mean, you have no idea what the hell is in there. You know, everybody dumps their stuff in there.…”

“the main risk is that, um, it's not that you waste the money with the acquisition, It's that you demoralize every, everybody else in the company because they see these new companies being acquired and they don't understand the value or they wonder, you know, why you might pay some new people a lot of money instead of…”

“I think it's honestly harder for some, Machine learning people than it is, uh, you know, for like a brand new entrepreneur who's, you know, just looking for an interesting thing to go do because it is a very different way for a machine learning team to do its work. And it's like been hard, you know, even for some of…”

“What I think we're starting to see right now is a, uh, the pendulum starting to swing back in the sense that there is a greater understanding that you really need a bit of both. You need that, uh, hugely powerful pattern recognition that we get from deep learning, but you also need the ability to reason about things…”

“there's a lot of tech people who come in To, um, life sciences, and it's like, we have that cell verbal. We are the smartest. We're machine learning. We're going to solve everything. And they don't respect the challenges of the other discipline. They sometimes don't even take the time to learn what the challenges of…”

“building in the open with open source is actually Uh, more part of the solution than part of the problem, because obviously control of power as you, uh, democratize it much more and biases you actually include in the process. People who are impacted by these biases, especially underrepresented populations, which is…”

“conversation is the future interface. And Google is already a conversation. It's just an appallingly painful one, right? You say something to Google, it gives you an answer in 10 blue links. You say something about those 10 blue links by clicking on it. You, it, it generates that page. You look at that page, you say…”

“the vehicle is only like, 15% of the complexity of the solution here. This is, by the way, a problem Or a realization that I think most robotics and autonomy companies have not yet had. Like you don't really realize it until you start trying to serve customers and suddenly have this really rude awakening that like the…”

“Doesn't matter how much money you've raised. It is going to be a rude awakening when you actually start charging customers for your product, because that is when you actually learn all of the things that you need to fix and all the things that are going to be super important to build.”

“Without being able to track that down, we'll never know if these models are generalizing or just kind of cleverly patching together what they know.”

“I do say these large models as well should be viewed as fiction, creative models, not fact models, because otherwise we've created the most efficient compression in the world. Does it make sense you can take terabytes of data and compress it down to a few gigabytes with no loss? No, of course you lose something. You…”

“And like, the problem is super simple to define. It's just like, um, predict the next word, the fat cat sat on the, you know, like, okay, well, you know, what comes next? Like it's, it's, it's extremely easy to define. And if you can do a great job of it, like, you know, then you get everything that you're seeing right…”

“Like if you're like such a public, you know, trying to present like one public persona that everyone likes, you're going to end up just being boring essentially. And people just don't want boring. You know, people want like the, you know, uh, want to interact with, uh, something that feels human, you know?”

“We just had a larger data set of, you know, human-like conversations, and we had this, um, you know, very kind of modest size open source model, uh, that's only six billion parameters, only trained on, uh, less than one terabyte of text. So like, 50 times less data than GPD three, and it still has this behavior. It's,…”

“it's, um, it's surprisingly good at just freeform, like kind of fluent text generation. So, um, you can tell it to like create a story or create a tweet or create a scientific paper abstract, and it does a pretty good job at that. And before that, whenever I talked to my, you know, NLP, like researcher friends, they…”

“this kind of token by token generation we're doing now is not an amazing format for reasoning because you have to like linearly, like do one, say one thing at a time. Um, so it's not really good for like making plans or comparing versions. I think to get a really smart application, you'll need to combine today's…”

“So it's always been clear to me that, uh, in order to work in AI, both research and product, you do not need to already have been in AI.”

“turns out through my experience, the best research scientists, also very good engineers, like very good engineers. And, um, And we've noticed like through DeepMind and OpenAI is just like the, the, the companies that made the most progress over like last six years or five years where people were, were companies where…”

“And I also don't think One LLM will just magically solve this problem. Uh, you need to build an end-to-end platform where users can correct the mistakes of an LLM, and that also means you need to design the platform where the incentive is right for the user”

“I think it's best to look for alternatives to the transformer, alternatives to like language models, that sort of radical directions, then Trying to improve them because there's so much incentive for the existing companies to do that.”

“I think the major challenge with, uh, Using things like reinforcement learning for, uh, trading is that it's a non-stationary, uh, environment. So you can have all this historical data, but it's, it's not a, it's not a stationary system and it's gonna like the markets respond to world events, these kinds of things. So…”

“And now today I think it's the dynamic is, is quite different because it's no longer academia's job isn't just to get things to work because, um, you can do that in other ways. There's a lot of resources going into, um, big tech companies where there's, uh, if you have data and compute, you can just sort of scale and…”

“I think we're getting to a point where along many axes, it's a superhuman or should be superhuman. And, and I think we should maybe define more of an objective measure of like what we actually want. We want something that's very reliable, is grounded. Uh, you know, I often want more statistical evidence when I speak to…”

“if you think about predicting the next word, It's, it seems very simple, but you have to really internalize a lot of what is going on in this context. What are the previous words? What's the syntax? What's who's saying them? And all of that information and context has to get compressed, and then that allows you to…”

“we realized that a lot of problems that Google solved with massive scale and user data, they could, in fact, solve with LLMs. So we use a lot of them for things like query rewriting.”

“ads sort of come, uh, you know, with elements of self-destruction built in. It's part for the course when you're doing it, it's always attractive to do things like show more ads.”

“If you're in the business of providing answers like ChatGPT was, um, ads just becomes a whole lot harder to do. Um, they're really, you're just, uh, you know, you're betting on the quality of answers”

“the observation that computer programming has now, uh, been completely disrupted. That for the very first time in the history of computing, the language of programming a computer is human. You know, any human language. And, and, uh, it doesn't even have to be grammatically correct. And it's fairly incredible that,…”

“The number of direct reports of CEOs are very few. And, and the direct reports of the people who are just learning how to manage first level managers are very large. It's exactly the opposite of how it should probably be architected.”

“I think ignorance is one of the superpowers of an entrepreneur, and you'll never get it again.”

“And I think by the way, if you make some money in software, you kind of have an obligation to like put it into hard tech and help some of the bigger things that might advance society, uh, which are tend to be more capital intensive and intensive.”

“When you can think about it, ah, models, ah, compute, everything is relatively ephemeral, ah, almost zero switching costs, and those are infrastructure, important parts of the infrastructure. For the industry. But if you're a company, whether you're a hotel chain, or you're a food chain, a technology company, it…”

“I mean, to me, I don't mean that metaphorically, like the brain very literally, very clearly, plainly is a computer in, in my understanding of the world. I also view the universe generally as a computer. Like we can solve pro like you can solve computational problems by arranging matter in a certain way, and then like…”

“if we are serious about exploring the universe and going to the stars, we are going to have to adapt ourselves to that environment. We're not going to export earth with us everywhere we go. And these bodies are great, but they're designed for this planet.”

“The alternative way that you can reduce risk is actually just reducing consequence, right? And we would argue that that's a much better way to reduce risk. Odds are stochastic, right? Even if you do an enormous amount of risk reduction on odds and you do all this engineering, something's still going to happen, right?…”

“the problem of nuclear today is like the Toyota Camry problem, right? Like we don't want to make Lamborghinis. We want to make a very simple, very cheap, very safe reactor that we can make literally tens of thousands of. Um, and actually that is going to make the cheapest energy in the world, right?”

“for a pure software and content business like ours, um, you know, and one where we didn't have like, you know, no major tech company was bearing down on us. We were able to kind of take a little bit of the slower, a slower road there, um, which was great, uh, developed a great culture and allowed us to, you know, not,…”

“most companies in any given era should at least consider it, and there's usually a time maximizing window where your best outcome is a sale within that window. It's like a 12 to 18 month period, usually where the company's worth the most it'll ever be worth.”

“Because if you're on the wrong side of this secular change, and you can't get to the other side of it, you should in fact, like, sell. You don't have good ideas about how to be on the right side of history.”

“Secondary is an intermediate option, which I actually don't think is always that great because it solves for some short term needs, but it doesn't actually, um, create a solution. And you see a lot of people from 20, 20, 21 still running companies five years later that aren't working.”

“There's a lot of successful rollups and tech enabled rollups, but at the end, all the products you're building are for the company you just bought. It can't really be actually applied to any other company.”

“This idea that the models, you just let them think for a week or whatever, and then they respond, it's, it sounds nice, and yes, the benchmarks look great, but it's not very practical when working because like, okay, you ask the model a question, and then you sit there for a week waiting for it to come back to you. I…”

“the model release cycle is, look, we're releasing new models, like, every two or three months at this point, and so a model comes out, it takes two or three months to push it to its limits, and then you have another model come out, and so nobody actually knows what the ceiling of capabilities are for these models…”

“there are some benchmarks where the models will just not improve if they have more inference budget. So I think a lot of factual, uh, factual, um, retrieval kind of questions fall into this category of if you ask a person when was Abraham Lincoln born and they don't know the date, they could sit there, they could think…”

“We're not seeing that with AI models today. They kind of, they're, they're born into a world for, and they exist for a very short context window, and then they just, like, disappear. And yeah, there are things that you can kind of do to, like, continue them, but it's very limited.”

“many people now that I know are hiring people more in their thirties, forties, fifties, because they're used to managing teams. And I think that transfers directly over to managing agents in terms of understanding the complexity of what to set up and the QA and everything else.”

“And then in terms of the capital intensive, like AI in a factory and also the foundry, and then you really need to tap either government funding or some sovereign fund, and also some very big capital. Uh, you know, there's some big fund that doing that, and they're really, the fund they've organized is basically…”

“you want to build a frontier AI lab, but you need to couple that with a frontier biology effort that can do the work of, um, of, of basically being able to Uh, understand and get the data that you need to actually be able to build these models, because unlike language models, where there's just like a lot of data out…”

“we have these systems that can basically kind of predict the next token and they can, you know, learn, World models from that. They can learn biology from the data.”

“at least for me, having grown up at Microsoft, having seen whatever four major platform shifts, uh, I sort of fall into that, uh, camp where a platform is defined by fundamentally its ability to create more value about the platform versus what's captured in the platform.”

“coding has worked so well that we now have to rebuild the IDE, right? I mean, it's kind of nuts to see what we launched is like, oh my God, I have these hundred agent sessions. I, the cognitive load, it transfers back to me as a human is so excessive that now I need a new UI. Uh, oh, by the way, I like the, the chat as…”

“the amount of work you need to do to prep the context layer, uh, such that your plan can execute in the most efficient way. Is where the magic is.”

“another acid test is you have an eval that's private. You're using a model A. Can you switch it to model B and, you know, climb up? If you can, then you're in control. If you can't, you're not in control.”

“unless we as an industry, uh, are very principled about ensuring That the benefits of all the stuff we're talking about are felt in real ways, uh, at the community level... then we will have permission. If it is not, we won't have permission.”

“there's a lot of systems where that's just not viable technically because AI today runs On the cloud, on someone else's infrastructure, on your endpoint, and just proxy is not always an option.”

“The hard problem is understanding if what I should do now. It turns out that in the case of AI systems, that is the hard question. Like, what is the engine that needs to underwrite these different actions and say if they're okay or not? And because We need to be able to understand what another AI system is thinking.…”

“I would advise everyone to assume that these models are coming anyway. The only thing you can do right now is to invest in these foundational controls that will stop the downstream effects of these vulnerabilities are going to be found in their systems.”

“We, we think that you can drive better win-win business outcomes with deeper alignment. And so by actually owning the companies and, and owning those customer relationships directly, we can drive better, better results. Software companies are wonderful. We partner with many of them, but when you're just selling…”

“technologists are very bad at hiring business people early on, at least in their careers and vice versa. Business people tend to be awful at hiring engineers. And so you end up with these mismatches on the early team.”

“if you feel like you are micromanaging, if you feel like you need, if you feel like, you know, you, you have to be involved in everything, I think that's a bit of a cop-out as a founder, because you're just like, I just need to be involved in everything. It's like, no, you, you probably don't have, The right people.”

“even if there's AGI, all that's left is inference.”

“the problem of why does agent decoding work so well, Sarah, is, um, of course you can verify the outcome, right? Uh, you can either say, hey, is the program compiling or are your unit tests, right? Does it work, et cetera.”

“Now, if you want to do these predictions, quite frankly, then the challenge is large language models are not made for this, right? The way how they, you know, generate just one token after another, essentially, in a sequence-to-sequence modeling. I mean, They're language models, right? And they do this phenomenally…”

“And in some ways, Bitcoin is full reserve money because you, you, you know, you, you kind of, there is no way to fractionally lend Bitcoin per se.”

“They need a medium A trust, a trust, a trustworthy medium where they can do that, where they can instantiate an entity, where they can store value in that entity. They can execute and arrange contracts, uh, that intermediate the work and the tasks, and that where all of it is real time mathematically, uh, and…”

“One of the engineers on our team was looking at a reported material, um, property and It was just sort of extracted values from literature, and it was really interesting to see the reported value spanned many orders of magnitude. And so you train an ML system on that, and it's like, well, the best you can do is model…”

“It's just like you can, you can, you can move in much larger macro actions. It's not just like, here's a line of code, here's a new function. It's like, here's a new functionality. And delegate it to agent one. Here's a new functionality that's not going to interfere with the other one. Give it agent two. And then try…”

“this is extremely well suited to anything that has objective, uh, metrics that are easy to evaluate. So for example, like writing kernels for more efficient CUDA, you know, code for various parts of a model, etc., are the perfect fit. Because you have inefficient code, and then you want efficient code that has the…”

“flipping, flipping bits and, and the ability to copy paste digital information is like, makes everything a million times faster than accelerating matter, you know? So, um, so energetically, I just think we're going to see a huge amount of activity in digital space, huge amount of rewriting, huge amount of activity,…”

“training neural nets and LLMs specifically, um, is a huge amount of code, but all of that code is actually complexity from efficiency. It's just because you need it to go fast. If you don't need it to go fast and you just care about the algorithm, then that algorithm actually is the 200 lines of Python.”

“I would say, I mean, the interesting thing is that with embeddings, it almost doesn't matter as much anymore, The, the AI doesn't really care what the, what the tree structure is, for example. The, all, all the AI cares about is that there's a snippet of text that has the, the context you need, and then it can retrieve…”

“If you're working with your, Your swarm of, like, a hundred background coding agents. You don't want to have a hundred chat threads. You want a Kanban board. It's, you know, the same as before.”

“Well, you know, fast forward to twenty-twenty-six, and what we see is, you know, there is obviously more availability of chips, but to build and operate these, you know, data centers requires people, power, infrastructure, a lot of these things that have a lot of bottlenecks. And so actually taking these chips and then…”

“there are a number of applications that, you know, on paper sound really interesting, like, oh, yeah, I could just rebuild Slack or you could rebuild Salesforce or could rebuild, you know, X, Y, and Z. I think, you know, the, It's not just the product, it's the way that's integrated across multiple services and systems…”

“I actually think the idea that actual production of code becomes not the bottleneck for, um, if you know what the spec is, not the bottleneck is like incredibly interesting, but I, I, I do think it overstates like how much of the overall software vendor problem that is.”

“If you're driving a level two system or a level three system or a level four system, For 99.9999, like three or four nine, identical. The difference is like the fifth or sixth or seventh nine on that is these like extreme corner cases.”

“The differences in the way it drives or feels are going to be more about like, what's the UI, the user interface of it.”

“And so if you put those at two extremes, one is communication platform, high fidelity with people, you know, the other is real time dreaming where everyone's an NPC. Everything in between is possible. And I think we see everything in between ultimately as being possible, and we may see weird product categories we never…”

“the big unknown to get to whole body reversible is the brain. Um, it's unclear, like, like the brain can withstand a lot of change and does withstand a lot of different types of damage or, or change with age, for example. Um, but it's unclear whether, like, what kind of injury the brain could sustain in the context of,…”

“One of the things is platforms are sticky, products are not. So no matter which software company you create today in the world of, ah, in the age of AI, or you created in the past, Products can be replaced.”

“the first, the first part of the safety of a product is that it performed as advertised. The first part of safety is performance.”

“I think this is, by the way, super under discussed that the people who tended to be the slowest adopters of technology love AI. That's physicians, that's lawyers, that's certain accounting types. It's, you know, it's, it's actually kind of fascinating. It's compliance. You know, it's all the people who always never…”

“I think you actually have the same problem in programming, where I think in the short term programming is verifiable, where you can look at unit tests, but once you get into real software engineering, like the unit, there is no unit test. It's like I deployed this and a million users used it for six months and it…”

“the big challenge that they ran into was, you're essentially just building two different companies, right? You're building a law firm, And you're building a tech company, and it's already really hard to, like, build product engineering, do AI, scale sales, and I think the big issue you run into if you try to do both of…”

“making someone program 20% faster doesn't make you build a product 20% faster. And so starting to think about, like, what is the broader infrastructure you need so these companies can develop software and product faster?”

“I find that having five, 10 customers is actually like a pretty representative model for what the entire industry needs.”

“I think, like, part of this is you can just allocate compute a lot more efficiently because you can defer stuff that the model doesn't have comparative advantage to doing to a tool that is, like, really well suited to doing that thing.”

“Just like automation is a better place to be than productivity enhancement. Like one of the challenges with our business, I think with a lot of the coding businesses is just like proving the ROI. Like, and there've been these studies that show like, you know, you deploy this stuff on real code bases. It's kind of…”

“Like, 90% of the cost down comes from technology and innovation, and 10% comes from cost of capital.”

“I would say that, um, memory is definitely like a feature that has been under, under-invested in, uh, by the field. Uh, but I would say that it is kind of difficult to invest in memory in this very, like, task-centric regime. Because if you have, like, A bunch of these, like, independent tasks, the amount of…”

“I think the challenge right now is that if you're building a wrapper effectively as an AI, as a service company, and all your wrapper does is enhance the capabilities of a model or put some guardrails around it, then your biggest risk is the model slowly expands into those capabilities and you're no longer in business.”

“The good news was there have been learning science papers written for 40 years back, you know, even before I went to high school that Talked about how kids could learn two, five, or 10 times faster. They just don't work in a teacher in front of a classroom model.”

“And if it's not real money, right. And if it's just like a, oh, a shadow account, right. Or fake account, it actually teaches dysfunctional behavior. Cause it actually teaches you to Lolo Yolo. Cause there's no downside.”

“Generally, when I talk to engineers that want to join startups, for example, and let's say they eventually want to start their own company, which is a very common, uh, profile, it's, in my opinion, it's like way more useful to join somewhere where they've already kind of got the commercials figured out and you can…”

“It also really changes how you think about the total addressable markets for some of these things, because if you're charging per seat, you're, you're really limited by the number of people working at the company. If you're charging per conversation or per some aspect of code written or other things, and really the…”

“what we saw in Silicon Valley is that they're amazing at solving problems. They're not always the best at identifying which problems to solve.”

“I think that it's very, very important to have private sector people going into government, doing tours of duty, Uh, versus just having a permanent political class, uh, which I think is, uh, is not necessarily great for the long-term, uh, interests of America.”

“a patient simply finding some clinical trial published in the New England Journal of Medicine, because it was mentioned on CNN or Fox News, and then going and trying to read it, especially through the lens of fear or hope, Is not necessarily going to result in a, in the most sort of constructive decision making…”

“But what I see is the single biggest Barrier to getting more, uh, agentic AI workflows implemented is, is actually talent.”

“So when we go look at this loop, the speed of coding is accelerating, the cost is falling, and so increasingly the bottleneck is actually product management. So, so the product management bottleneck is now we can build what do we want much faster while the bottleneck is deciding what do we actually want to build.”

“AI might be the most transformational economic cultural force of our lifetime. And I believe that if the country or the ecosystem Which winds up getting ahead is going to have these cyclic effects, right? Like you're going to, you know, you're going to power productivity. You're going to have drug discovery. You're…”

“One of the things I saw when I was at Twitter was how easily you could inject cultural bias in your algorithms. I have so many stories about how if you pick the right kind of Twitter accounts, which then feed into the trending algorithm, which then feed into Twitter moments, and then with every journalist or editor…”

“if you have a model up on hugging face and somebody downloads a final gigabyte file and the thousands of students and researchers just kind of pounding away on that, I think there's a good chance they're going to find issues a lot better than a very small safety team, uh, inside a large lab.”

“Like I would never hire data scientists when the first three people in a company. And I, and I say that because I used to be a data scientist. Like data scientists are great when you want to optimize your product by two percent or five percent, but that's definitely not what you want to be doing when you start a…”

“product managers are great when your company gets big enough. But at the beginning, you, you should be thinking about yourself about what product you want to build. And your engineer should be hands-on. They should be having great ideas as well.”

“I think one of the things that people really underestimate is how it is, how complicated it is that you can't just synthetically generate it.”

“As long as you pick a category that I would say is kind of big enough to be ASI complete, um, I think, and this is kind of our approach at Reflection, is it makes a lot more sense to be focused and co-design those two things together, the product of the research.”

“I don't think it's guaranteed that a big lab can, you know, buy their way to, uh, to the end user because the fundamental problems of your, you know, research team being far away from your product team will still be true. And, and the company having, you know, a hundred different focus areas will still be true. So I…”

“there was a period of time where, you know, yeah, if there was no sort of SaaS application for I don't know, whatever, like, time tracking or expense management or something like that. You could build a standalone thing for that, and it would get, like, very rapid uptake, but as soon as you start to get the, sort of,…”

“I think you just learn that unless somebody is really taking care to keep the entire system in their head and is an effective technical steward of the architecture, that things just evolve and the sort of the entropy of the software takes over and slows down your rate of progress to zero because nobody can, can get,…”

“Yeah, I guess the way I define my metric for when things start to get really interesting from a societal and cultural standpoint is when we've passed the economic Turing test, which is if you take a market basket that represents like, 50% of economically valuable tasks, and you basically have the hiring manager for…”

“AI is not complete without spatial intelligence because, uh, um, The humans interact in, um, in, uh, three D worlds and in the digital world, we need all kinds of interaction”

“if I was a number one or number two in a market and I was a startup, I'd consider merging with the other party if there were the two main startup players, because the real threat will be fighting the incumbents.”

“a lot of the companies are started as incubations by biotech VCs. So they load up a company with forty million dollars. They buy 40% of it upfront, whatever it is. And then they kind of have to make it far enough that they can get, um, almost public market money effectively. You know, a lot of the crossover funds then…”

“On the web, even if you don't have good semantic understanding, there is so much that you're going to learn from people's behavior because, you know, you have a billion people, you know, coming and using your product. In the enterprise, you don't have that luxury. So you have to sort of like, you know, make, make up…”

“businesses are actually a lot more, more interested in, not in that, but in actually thinking about how they can transform their company with AI, how they can take specific business processes, you know, where they're spending a lot of money, uh, and how, how do they bring automation in that with AI?”

“So the right recipe for me, like, you know, if I had a choice, I would, I would actually start both the motions simultaneously. Like, I don't want to actually Say that, look, you know, for the first three years, I'm, I will actually focus, you know, uh, on just being PLG and then bring the enterprise states later,…”

“neither of us likes learning languages. We actually are not language lovers. We don't like learning languages. And I think because of that, we made a Product that works for the average person, as opposed to for people who are obsessed with learning languages.”

“whenever we give you an exercise, the right thing to do is to give you an exercise that you're about 83% chance of getting it correct. Uh, turns out that maximizes enjoyment.”

“Secondly, I think the overall goal for OpenAI is to create an AGI that can make new scientific discoveries, and we kind of felt that a prerequisite to that is to be able to synthesize information. You know, if you can't write a literature review, you're not going to be able to write a new scientific paper, so felt very…”

“I think if you have a very specific task that you think is so different to anything that the model was likely trained on and you try it a bunch of times yourself and you've tried a lot of different prompts and it's just really not good at it. So maybe it's, um, genetic sequencing task or something that's just so out of…”

“There is no way to isolate the AI from the HI. There's no way to isolate one piece of data that's uniquely powerful, and that's one of the reasons that I think is limited innovation as well”

“It is much, it is much easier to know what a mine is going to produce 20 years from now than it is to know what a SaaS company's sales volume is going to be 20 years from now and how it's going to be priced, right?”

“I think that the real scarcity is good quality or deposits. Like great projects don't have problems getting funded. Whoever owns them. Great, great projects, uh, have lots of suitors of people who want to buy them. Uh, the problem is there just aren't very many great projects.”

“You don't even know that you have super intelligence without having evals for everything. Cause it's like, you sort of need to understand what is the human baseline and like, what is good? It's like grounded in this like understanding of human behavior.”

“It does seem structurally more efficient for work to trend away from the variable cost of like doing it repeatedly towards this fixed cost of how do we build out the evals and the processes for models to do this themselves.”

“all these CMIOs and CIOs are on WhatsApp groups every single day and talking to each other. And if you screw up with one of those health systems, maybe two of those health systems, you're kind of done for like a couple years, probably. You don't get another shot on goal for a really, really long time.”

“We had a research result Um, a couple years ago where we showed that if you leverage vision language models, then you could actually get the robot to do tasks that require concepts that were never in the robot's training data, but were in the internet.”

“I feel like actually people underestimate how much intelligence goes into motor control. Many, many years of evolution is what led to us being able to use our hands the way that we do. And there are many animals that they can't do it, um, even though they had so much, so many years of evolution. And so I think that…”

“I think that data can have a lot of value, but I think that by itself, it won't get you very far, uh, and I think that there's actually some really nice analogies you can make where, um, for example, if you watch, like, an Olympic swimmer, swimmer race, uh, even if you had their strength, uh, just their practice at…”

“human developers are expensive because, uh, there's limited supply, uh, agents will have infinite supply that, that will only be limited by the amount of, uh, compute capacity, uh, GPUs available in data centers.”

“So I view the distinction between alignment and safety, um, as alignment as being a sort of subset of safety. Obviously you want the value systems of the AIs to be in keeping with or compatible with, um, say the US public for USAIs or for you as an individual, but that doesn't make it necessarily safe.”

“So you can restrict their intent, which is what deterrence does, but I don't think you can reliably or robustly restrict their capabilities.”

“The learning cycles of doing things at the same time is completely different than if you actually force yourself to think about the essence of what you're doing and what's reproducible, what has to be different in each case.”

“most, um, successful entrepreneurs who've done, you know, kind of like unexpected things also secretly worship at the altar of chance, because they also realize that at the end of the day, their explanations of what happened. Do not suffice for actually what happened.”

“The price of diesel, whether it's renewable or not, is the price of diesel. The price of carbon when we started was 50 bucks a ton, when we ended it was five. And When we started, the U S was energy dependent. And when we were done, the U S was cushing with energy, you know, liquid fuels and oil and gas. So in the end,…”

“in the science driven, technology driven space, I think adjacencies have their own problem, which is the risk of commoditization, which is a risk they usually don't take into account.”

“Second, investors don't properly value platforms because they don't give assigned value to the correlated option value of one program that becomes de-risked based on three other programs, and so you don't get the credit And you get the, the, the deduction because it's now all of a sudden overly expensive and overly…”

“previously the virtual biotech was a concept that was very much in fashion, right? Folks found out just In reality, when you try to do this, even though it looks really good on paper, it's incredibly slow, right? So then folks tried the other way, which is let's just fully vertically integrate and just own everything.…”

“I think in the home and most consumer app applications and even most industrial applications, um, safety is still critically important, but the bar that you need to reach or the functionality that you need to re reach or the constraints you can put on the system, uh, enable you to launch a product much more quickly.”

“If you have a couple of good engineers, they can adapt and do all those things when, when sitting next to an LLM and some good coding tools.”

“If you are using these tools and you don't think that you can take basically AI and apply to X industry and transform the entire industry, I don't think you're thinking ambitiously enough, right?”

“these models can't just one shot all of these really complex legal tasks or, or in these other domains like tax and provide other professional services. And so what you have to do is you have to build a platform that is kind of constantly expanding and constantly collapsing. And so what I mean by that is you need to…”

“you need to build productivity tools, and what I mean by this is Things that are useful for the highest amount of seats, right? And then you also need to build things that are streamlined vertical workflow from start to finish, right? And what you can do is you can take those streamlined vertical workflows and you…”

“On the productivity side, the minimum viable quality of that output can be lower because you're selling seats, And at the end of the day, there are multiple people reviewing it, right? Um, and so what you want to do in that state is have just show your work, right? Like that is the most important thing. You want to…”

“we found that no matter what, you eventually have to get the trust of the industry. And the best way to do that is to actually go after the hardest people first.”

“It is not job. Displacement is task displacement. And I think that's a super important distinction because getting rid of those tasks does not mean the legal industry falls apart. It will evolve.”

“one of the lessons that I definitely want to keep is I do think you should do every single role For a certain amount of time before you hire for it. Almost all of my mishires were because I did not understand what that role was.”

“I think that it is very overvalued how much value you get from asking people that you really respect, very specific questions, I think is overvalued. I think spending time with people that are incredible at whatever they do is undervalued. And I think it's because the latter has like a nebulous effect.”

“So one thing that I look at is you can look at this by industry and you can look at it by task, which is how expensive is the token? Um, and I, I, I've come up with like different ways to call this, but basically if you look at something like that share purchase agreement or a merger agreement, or just legal in…”

“One like mistake that entrepreneurs and investors make that I have made is like you look at something and it's not working and like the issue is it is like a technical issue and then you like assume it's not going to work. But I think in AI you have to like keep looking again and again because stuff can begin to work…”

“and for us, actually those things help, but they're actually not the biggest difference maker. So in our use case, the type of intelligence that matters the most, um, We would probably describe it as instruction following.”

“the use cases that emerged, like you have to have those two qualities. Like it has to be able to be something that can be rolled out slowly and doesn't have the perfect off the bat, but it's already providing value, right? Like I think coding agents is like a good example of this where, um, you can just like section…”

“I think eventually like for any company where model quality really matters, unless you kind of train your own model in the end, like, I feel like it's going to be hard to sort of defend the fact that like, You know, you're, you have a better solution. Cause like otherwise, like what's your moat? Like if you don't have…”

“So if you're serious about software, you build your own computers. If you're serious about software, then you're going to build your whole computer, and we build it all at scale.”

“Startups. We, we know how to like build products. I mean, we know how to raise money, hire people, build products. We know how to sell it. Sometimes we know how to market them, but I don't believe we know how to take care of our customers, especially in the B to B. I, I think managing customers both at scale, when you…”

“I do not want a bunch of, could be conceivably very talented people, like between me and the founders that I, you know, owe support and partnership and work to, because I don't have any of the context. I don't care as much as I do. Um, and I think it's like, uh, You know, not a great experience when there's like four…”

“to trust the output of an LLM to send an email to a million dollar customer. It's like, you know, when you present that to a, to a CSM, they're like looking at it and they're like, they're questioning every word. You have no idea. They're questioning every word in that email, yet when they're writing their emails,…”

“because it turns out kind of managing, uh, an AI is extremely similar to managing actual software developers.”

“Well, I, I think all language models are quite sensitive, uh, to, to prompts, to the way that you present data. They all have their own individual quirks. The way that you talk to one might not work for the way that you talk to another. And so when you're building a system like a rag system where there's an external…”

“There's just no way you can do that without intervening on pre-training. You can't like fine tune or post train Japanese into a model effectively. And so you have to start from scratch.”

“Taking those models and trying to, um, fine tune them It's just, it's not as effective as building it yourself and you have much fewer levers to pull, um, than if you actually have access to the data and you can change the data that goes into that process.”

“the advantage of that is that once kind of the, the proof is complete, uh, then you, you, you know, the, the machine would give you a signal back to say, yes, your proof is correct or not. And so we could search for kind of correct proofs. Uh, once we find a correct proof, we can learn from it, uh, and get a better…”

“But if you instead relied on a formal system to check everyone else's work, then You could do a little bit like in astronomy where you could have, um, an amateur kind of living in the middle of maybe nowhere and you, you've never met. And then you wouldn't have to trust him. Like you could trust in some sense the, the,…”

“These new data centers we're creating are not data centers. They don't, they're not multi-tenant. They tend to be single tenant. They're not storing any of our files. They're just, they're producing something and they're producing tokens.”

“Um, and then they're a little bit different, but if we're talking about full autonomy, maybe there's second order differences, But the first order of complexity is still there. You can, ah, like the, if you think about, you know, the core, the heart of the problem of, you know, building a generalizable and safe driver,…”

“The complexity is in the long tail of the many, many, many nines. And you don't see that if you go, you know, uh, for a prototype, if you go for, you know, a driver assist system, uh, and this is where, you know, we've been spending all of our, that's the only hard part of the problem, right?”

“there's research suggesting that the non-enforceability of non-competes in California has actually been one of the drivers of so much innovation, because it's allowed this more organic diffusion of ideas in a way that's just been really, really healthy and Led to a lot of creativity and great outcomes.”

“GitHub can't do this because GitHub doesn't know who you've promoted. They don't know who did well. They don't know who you've had to let go of for performance reasons. Salesforce doesn't know that either. And so, like, I'm sure there's gonna be really cool, like, code quality evaluation tools built into many of these…”

“And then I would say also one last dimension of it is you benefit a ton from like the transfer learning between the different tasks. And in AI, you really want a single neural nut that is multitasking, doing lots of things that's very getting all the intelligence and the capability from. And that's also why language…”

“That's the thing with AI. I feel like a lot of them, a lot of these capabilities are just kind of like prompt away. So you always get like demos, but like, do you actually get a product? You know what I mean? So, um, so in this sense, I would say the demo is near, but the product is far.”

“It takes effort, but it's effortful, but it's also kind of fun. And you also have a payoff of like, You feel good about yourself in various ways, right? And I think education is basically equivalent to that. So that's what I mean when I say education should not be fun, etc. I mean, it is kind of fun, but it's like a…”

“Well, so you can think of model performance as some function of training compute times some function of inference time compute. Now those are specific functions that are just scaling law things that you can like model, but the, the general Way to think about it is some function and some function, and then you would…”

“No, I think if you can, the leap to doing all use cases is small. Like you can build a UI builder and then it's like a normal UI builder, or you can build a true great UI builder driven by AI with some added features for that vertical. But then you can do the same thing for all the other verticals, just add the…”

“Inference is, is one of those, uh, workloads that today it's, you know, fifty-fifty maybe of training in Inference, but, uh, in order for the math to work out, Inference workloads have to dominate, otherwise all this investment in, in these big models isn't really gonna pay off”

“I think it's a lot more restrictive to start with, let's make something on brand, and then let's make it work, and a lot more costly. Um, so trying to, like, invert a little bit how, uh, how we produce, like, good creatives and good content.”

“to accelerate that, that sort of progress, you, you want to start sort of really exploring what's the reasoning the model has, and by making it more redundant, more logical, by iterating more on these kind of ideas, you could imagine, um, generating a, a very small program, right, that runs slowly with the language…”

“If checking that something is correct is easier than creating the solution, then we're in business because the language models will be able to evaluate their own samples more accurately than to generate them. And then we have a sort of reinforcement learning loop because we can reinforce the ones that seem more…”

“what we've learned is that, you know, this is gonna be a really difficult product to just kind of release out there in a horizontal way and hope that everyone just figures it out, right? Even though there are so many different applications, you can apply it to almost any use case, any industry. I think that the gap…”

“But in, in AI companies today, models are such a big piece of the product experience and that the. Uh, thus the like model design. And I do mean that in the aesthetic sense of the term is quite important and often comes from the founder. And I think an example, a bunch of examples of that could be, um, mid journey,…”

“From like a thesis perspective, I, I strongly believe in like an iterative design approach. We really don't believe on spending a lot of time, like just, just doing research and analyzing. We spend a lot of time on just testing, building the testing here.”

“Francois definition, which is the one that I think is the right one is, uh, this definition that general intelligence is a system that can effectively, efficiently acquire new skill. That's, that's it efficiently acquiring new skill and being able to solve these open-ended problems with that ability.”

“fine tuning in many cases doesn't work because, uh, you need a lot of data to see the results and there are still hallucinations even after fine tuning.”

“Which actually, I think that's even, you know, going back to the YC, the cult of YC, I'm starting to realize that is actually one of the core things that we've always believed that extremely technical people Uh, those are the people who probably should be, like, the Zucks and the Larry and Sergeys of the world. Like,…”

“actually the, the mega tech companies have such unassailable moats, and they make so much money that they're actually somewhat incapable of, like, stomping anymore.”

“One of the things that every, you know, all the model developers understand well, but the enterprises understand super well is that, um, you know, not all data is created equal and high quality data or frontier data is, is, can be, you know, 10,000 times more valuable than just any run of the mill data, uh, within an…”

“I think in, in all branches of AI, we become slaves to our metrics, and you say, I did this accuracy on this benchmark, and this accuracy on this benchmark, and in the real world, sometimes it doesn't necessarily matter, and these benchmarks are extra terrible in audio, um, just because the field is so new. And so…”

“music is done now, sometimes painfully, but only in service of the final product. Um, and I think when you open this up to people, Um, sure you definitely care about the final product about what the song sounds like on the other end, but you also really cared about the journey and that people will really enjoy making…”

“today to train these large models, you, you need all of the GPUs co-located because there is enough, um, Uh, data transfer between different chips, right? Between your nodes. And there's a physical constraint on that in that you need to get that much power and to a data, data center.”

“Devin is very much not the one who is going to be deciding what to do. If that makes sense. Um, and so, you know, we very much think of Devin as you provide the precise formulation of, of what you want built or what you want done, and Devin is the one that is doing the thoughtful execution of that, right?”

“there is this pressure to do things fast because AI is so fast, and the fastest thing to do is, oh, let's take what's working now and let's kind of, like, add on something to it, and that probably is, as you're saying, more general than just image to video but other things, but sometimes it takes taking a step back and…”

“instrumentals in music are actually very easy, uh, to get or to make. Um, you know, there, there's a wide variety of like quality, of course, but, uh, generally instrumentals in a song, like if you hear a song from Taylor Swift or whoever, a rap song, Those beats or those instrumentals are fairly easy to make and easy.…”

“And I think the things that we see making it into production and, and, and informed, um, a lot of the development of laying graph, um, is, or is something in the middle where it's like this controlled state machine type thing.”

“And, you know, what, what's crazy about it though, is I think the most efficient businesses through markups and through software optimization can actually direct, like drive pretty healthy margins. Um, and still have these really aggressive consumption, um, contract.”

“Here it feels like things are ramping really fast off of products that are a couple months old, which sometimes suggests that there's not defensibility.”

“So I think that we get these new glimpses at interaction patterns that are really cool. And the natural inclination is to extrapolate and say they're going to be useful for everything. And I think that they have like sort of their role and, um, it doesn't mean that they're going to be ubiquitous across every…”

“With other search technologies, this is again, this is the wrong search mode. If you're searching with keywords and just not finding The relevant information, because the embeddings, the, the contextual space in which these pieces of text, documents, or images live is in vector space, in high dimensional numeric space,…”

“Our research actually shows that when you do this well, we, you very rarely need keywords alongside embeddings, but getting embeddings to perform perfectly is, is actually, it could be quite intricate.”

“unless you have the research team and the, the AI experts that know how to fine tune, you might actually make things significantly worse. Okay. So there is, there's nothing that says that more data is going to make your model do better. In fact, it oftentimes gets, uh, regresses to something significantly worse.”

“No code is like coming from this kind of like power user developer angle of wouldn't be nice. Everybody can modify this underlying software they use every day. That's why I go there. It wasn't coming from the angle of the knowledge and data wants to be in one place, right?”

“Language prompt engineer language model sort of make everybody like a real-time machine learning engineer.”

“now the new technology happening is AI language model, and if you build more with it or just think more with it, language model wants information to be one place. Once the endpoints to be connected, so it's easier to, is it hard enough to like a conversion language model to do what you want, but imagine top with…”

“now I feel like people are both building great products as well as bundling them, but also they're much more aggressive about saying it can be 80% as good. It could be 50% as good, but I'm going to have a bundle and that's HubSpot and that's Rippling and they have very high quality to their products. It's just they…”

“I think there's bundling of distribution and bundling of information. What you're describing to me is more of Microsoft, more like bundling of distribution. Language model wants the bundling of data, bundling information.”

“unlike a bunch of the past generations of software startups, we're seeing AI startups have substantial compute costs right out of the gate, and that that's putting a bunch of pressure to build monetization engines faster.”

“the world is littered with the corpses of, you know, the companies that tried to serve SMB and enterprise at the same time.”

“Well, look, I mean, I think the, the exciting and frustrating thing about LLMs right now is that you can get an LLM like agent thing, like a sidekick-esque thing. You can get it to like 75% in like 10 minutes. And then it's like this brutal hill climb to get it to, like, 95%, like, over time, right?”

“yeah, you know, that's its job, right? You know that every time it emits a token, it's hallucinating, right? It's just that you like some of the hallucinations more than the others, right?”

“If it actually works, the strategic brilliance of that is they instantly have the world's largest app store because the world's largest app store is just the web, right?”

“the fidelity of today's systems far less than 90% for each step. So I think this is the issue that everyone building agents in that way is, is encountering is like, you know, you have compounding failure.”

“and if you fine-tune a, a, a medium-sized-ish model, sometimes it loses the ability to, to do effective, uh, in context learning, because I think the intuition is, it's devoting more, more of its parameter space to, kind of, like, memorizing the, the training set so it can do better, kind of, like, uh, rote completion…”

“every way of computing, you suddenly have this burst of, well, this thing is really working. Let's try five other things that could work potentially better. And then, you know, I, it feels like 90% of the time it collapses back to the original thing that you just keep scaling it or whatever”

“I'd say generally nonprofit governance is not every organization, but as a class known to be kind of a Right. Because performance is hard to measure objectively. And so it often ends up more about politics and specific relationships and status games than outcomes.”

“Assuming that the model should be well behaved is, I think, a wrong assumption. You need to make the assumption that the model should know everything. And then on top of that, have some modules that moderate and guardrail the model.”

“I think making a model smaller is definitely a way to make agent work. Uh, because one, one problem you have with agent that very quickly is start to, if you run an agent on GPT four, uh, you're going to run out of money very quickly. And so if you divide by a hundred, uh, the, well, the cost of compute, well, you, you…”

“The reason neural networks of the time weren't good is because they were too small. So like if you try to solve a vision task with a neural network, which has like a thousand neurons, what can it do? It can't do anything. It doesn't matter how good your learning is and everything else. But if you have a much larger…”

“Kind of classic cliches in the cybersecurity industry, like the cybersecurity skills shortage, like America needs, you know, to bring back the draft and make everyone get a security certificate or something. Okay. Like, you know, that you have like, 90% of a human that you can use for like a penny and a half, right?…”

“Take a step back and just, like, try and build an incredibly useful thing that everyone should buy. Stop thinking about the Gartner categories and, you know, whatever, CASB, UBA, SIM, whatever, DNR, something, something, something, like, stop, like, trying to, like, look at, at this, like, big, like, and you see these,…”

“early in markets, like when a new technology shifts and disrupts the whole market, you actually want to just do the easy stuff, right? Why do the hard stuff? There's so much low hanging fruit. Why don't you just go after the stuff? It's super easy. And my, my sort of advice to founders generically on this stuff is…”

“Like you're about to experience, like, like, like the analysis of these neural nets is going to look like systems biology, right? It's going to be like, go in and like, try to back, figure out a thing that you didn't design, my friends.”

“The real advantage is the end-to-end journey from the first line of code to the deployment, to getting crashes in production, to making edits, to all of that feedback cycle, which typically in most cases is divided between GitHub and VS Code and AWS and all of that. And the real magic of Replet is putting all of that…”