AI model

“There are agencies that are selling data, packages of data that are ready, so you can get them and train your models. But the question is, if everyone takes that exact data set, then the quality that every single company is producing is going to be the exact s…”

“I think there's a very real sense in which models today are not intelligence limited. They're eval limited. They can actually do much more and be much more correct on a wider range of things than they are today, and it's really about sort of teaching them.”

“So we had to spend our entire roadmap on that. We had to slow down all the other grand ideas we had. And I'll tell you, this was not always easy. We would look at other teams building the newest AI models or advanced photo filters, and we'd get a little bit je…”

“You know, models are trained on the distribution of data in the middle, but great, great decisions and new creative solutions to problems always fall outside the distribution of what's been done before. Otherwise, they would have been done already.”

“If you optimize the model for this behavior, like, you kind of don't want to, like, brain damage in, like, other areas of intelligence, or, and this is happening, like, all the time in every lab and every, like, research team.”

“I think the future is really gonna be incredibly smart, broad based models. That are fine tuned and tailored with company specific or use case specific data so that they perform really well on company specific or use case specific things. And you're going to m…”

“The models are good enough to do it now, and every, every study out there that's ever been done seems to show that when you have, you know, classrooms is still, classroom, like education is still important, but when you combine that with personalized tutoring,…”

“And so that, those tasks where you can almost model it to a translation task, definitely going away, but knowing how things work under the hood.”

“Instead, what I would do is I would chop things up into bits, and you can spend basically, you know, the same amount of time specifying things overall, but chopped up more. So you're specifying a little bit, you're getting a little bit of work, you're specifyi…”

“One lens through which you can view this company is they spent 20 years basically building the mother of all data foundations for every important institution in the world, and Guess what's very valuable now that AI models are out is proprietary data that isn't…”

“We try to automate as many things as possible. We use even AI models for that”

“Like there's still set, we call it overhang, right? Like the Delta between what the models and the products can do and how it's been, they're being used on a daily basis, huge overhang. So that's, we're still like a very, very strong necessary role for product…”

“What skills are going to be valuable in the AI era? One big group of skills are the skills of managers today. They're human managers tomorrow. Everyone's a model manager right now. AI is not like right now management skills are not broadly distributed because …”

“And if that continues, you know, in three years, we'll have a thousand X smarter models for the same price.”

“If the model is in a box trying to improve itself, then it could go completely off the rails and have these secret goals, like Resource accumulation and power seeking and resistance to shutdown that you really don't want in a very powerful model. And we've act…”

“I don't expect there to be a wall in terms of models ability to improve themselves if we can give them access to the ability to be empirical.”

“And also the models deteriorate in value fairly quickly as an asset class. And so you need just a lot of scale to make a return on the investment for a model that deteriorates in value so quickly.”

“And I think that almost always when you have a model making a poor decision, if it's a good model, it's lack of context.”

“The way to appeal to an AI model ideally is the same way that you would appeal to a real user because the model is supposed to proxy the interest of the user and nothing else.”

“One is sort of working backwards from the model capabilities, and that is much more than science where I think you really need to look at what tech do we have available and what is like the most awesome way to product productize it. And if you apply to some so…”

“The benchmarks are increasingly saturated. So really you need real world scenarios where your product or model is not actually doing the thing it was supposed to do. And the only way you get that is by shipping because you get back to sort of use case distribu…”

“But really what's happened is the models have gotten so good that the generalists are no longer needed. Like what they really need is experts, experts across every area that the models are focused on.”

“And I think that that road to improving models will last for as long as there is anything in the economy that humans can do, which models can't, and be a huge portion of what the future of work looks like.”

“If the model is the product, then the eval is the product requirement document.”

“I think it's gonna open up a lot of this democratization of accessing these models and building incredible things. Cause you don't even need to do a lot to get the most sophisticated outcomes. Increasingly. I don't think you need to do a lot of this heavy duty…”

“There are certain domains where it does really well, and, you know, I definitely like, oftentimes will, you know, ask an AI model about a legal question now before I call a lawyer, because I find it's not replacing my call with a great lawyer, but it does info…”

“Today, you take a model And run it through a video of a couple of office rooms and ask the model to count the number of chairs. And this is something a toddler could do, or maybe, maybe a elementary school kid could do. And AI could not do that, right?”

“It's not great to just give it access to your drive and say, reason all of this knowledge base. It actually does a very poor job understanding importance of the past and putting weights on stuff.”

“I think one of the things that's going to happen in the next few years is that the models are actually going to become increasingly differentiated because of the personalities and behaviors That the different labs have and the kind of objective functions that …”

“In my head, I probably bet that within the next one or two years, yeah, the models are going to automate 80% of, you know, the average L six software engineer's job. But it's going to take another few years, do you move to 90%, and another few years to 99%, an…”

“One of the things I often think about is that there's a, it's almost like there's an art to post training. It's not purely a science. Like when you were deciding what kind of model you're trying to create and what it's good at. There's this notion of taste and…”

“The agents are going to get better. The models are going to get better. They're not going to need me to extend the context. They're going to do it themselves.”

“Like so far we've trained these models with, you know human generated data publicly available on the internet, but we are running out of human generated data publicly available on the internet to train the models.”

“You can't, you just can't, we can't mock up all the states, you know, and you can't theorize and you can't even make like a clickable prototype where that you sort of have to use the actual models underneath and you have to sort of see people try it out with t…”

“I think humans still provide a level of common sense that the models don't. And there's like a thousand moving pieces to any product launch. Some of them are very small, but there's always a lot that could potentially go wrong. I think the model doesn't always…”

“I think for a lot of knowledge work tasks, probably sometime we'll get to good enough. And once you get to good enough, then you can optimize other things. Like they run locally, they're cheaper, they're faster.”

“My take, it's very clear, at least empirically that models are getting better at coding at some exponential rate, right? And I don't think that's changing. Now I'm not that impressed with the progress in any other domain. It tends to be like, I don't think the…”

“My experience and my feeling is that we will look back in a year and say, We actually have a lot more work to do. Humans have a lot more work to do. Even as models get better at doing work.”

“We have no theory of what human intelligence is. We have no theory of why these models work so well. We have no theory of how much better they will get. So we're all just kind of vibes forecasting as to what will happen.”

“So distribution of an adequate product when the field is basically commodity distribution on brand become a big deal.”

“Because an opinion based thing on highly innovative, differentiated stuff, it doesn't have a model for that. There's not that stuff in there. You gotta, you still need the very version one to be done.”

“One thing that I think all models suffer with right now is just, they usually increase complexity. If research is listening at any company, please make the models better at deleting code.”

“Having the most advanced AI and the access to the best models in the world won't determine whether your organization succeeds.”

“People who feel diminished now will only feel more diminished in the future. Probably.”

“I think fundamentally, you didn't ask me this, but there is this question in the community of, do we still need PMs when the models are still capable, when engineers are leaning in I think the role of people who are user centric, who go into the details of und…”

“There was a world where the models were coming out every three or four months. We're down to a number of days between models”

“I think you should use the models as much as possible for writing as reporting, and in as much as you think with writing, as I really do, and I think a lot of people do, you should not use it. You know, you shouldn't replace your thinking with it.”

“Some pieces of our craft are actually getting abstracted by models being able to do it really effectively, maybe better than individuals can. And your craft moves from being able to do that very specific task you did in the past to Now applying it to some othe…”

“We've seen one thing, Lenny, it's that the Ability for models to do new thinking out of distribution thinking is still really limited, and I don't actually take the point that some of the new thinking in math is actually representative of new thinking in domai…”

“The sort of concept of the model that's too dangerous to release it kind of conflates marketing inference capacity, and then also economic considerations, like, do you want to externalize your competitive advantage or use it to make yourself better?”