GPT 4

“There's sort of a post GPT-IV era, and then there's a more recent, like, post wide adoption of coding agents era, and then probably soon there's going to be, you know, additional eras, and things are going quite a bit faster, and development is going, you know…”

“When we got to GBT four, which was back in 20, 23, I think that very quickly after that, there was going to be much more disruption in software business being up for grabs right, right away than turned out to be. And the thing that I think I was wrong about a …”

“I would say we got real conviction with GPT-IV. Not even 3.5.”

“It clearly wasn't an AGI. It was lacking something, but just if you'd describe your criteria for AGI two months prior, it probably would have been compatible with what GPT-IV was”

“And this I think is like the fundamental difference between reasoning models and the four ones and four O's of the past where these models could not think. So you kind of had to put them in boxes with a predefined set of state transitions. Whereas here we have…”

“So we just gave it away for free and that ended up being totally revenue positive and retention positive because it just provided access to the tech.”

“A few weeks or so after I joined OpenAI GPT-IV had finished training, and I remember trying it out, and it actually It didn't impress me at all nor anyone else that week, because it kind of didn't work, and it's because we hadn't figured out how to post train …”

“It wasn't until I played with GPT-IV that it became really, really real. So, you know, we, when we got access to it was sort of like a proto-ChatGPT-like interface. And my co-founder Ivan and I both got access, and it was just immediately clear, like, I would …”

“We looked at the cost of a GPT-IV level or equivalent model. We looked at that a year or two ago and basically in 21 months, it went from like 37 bucks for a million tokens to 25 cents. And so, you know, pricing dropped by a 150 X in 21 months.”

“The cost of actually a token on GPT-IV has, you know, gone down a hundred X since the model was released.”

“And so I was a red teamer for GPT-IV and I was using it like nine months with me for release with August. So GPT-IV came out in March and I was using it in August.”

“We'd gotten the demand from him to say, man, challenge to say, hey, I'll believe you have general purpose, you know, and that's not AGI, but it's like a good general purpose functional technology. If without special training, you can pass the AP bio exam at a …”

“So I think that we're seeing a plateau in performance for consumer use cases. And the reason why I think this is like a misnomer or something that people actually shouldn't pay attention to is we don't need them to be better for consumer use cases. Like, I fee…”

“Back when I'm Googled a Gemini one when I ultra and needed a number that would say it was better than GPT four. And Like, constructed I think never published, like, chain of thought examples, 32 of them in every topic in MLU to run it, to get the score.”

“The, like, one fact on that is that you can get intelligence at the level of GPT-IV for over a hundred times cheaper than GPT-IV was at launch right now.”

“In 20, 24, the cost of GPT-IV equivalent models, if you look at a million tokens, it came down over a hundred X. You know, so many of my team did this analysis to show that.”

“You know, we've also had YC companies where, ah, they had something that beat OpenAI, ah, you know, GPT-III.V and they were doing fine-tuning with RL, but then, ah, yeah, GPT-IV and then, ah, GPT-IV came out and, ah, you know, basically blew their fine-tuning …”

“The pre-training part in that timeframe was mostly about making things cheaper. Not making things better.”

“We actually got a demo of GBT-IV in November of twenty-twenty-two before it was publicly released in March, and it was one of the most astounding pieces of technology we'd ever seen.”

“So I think I'm going to claim here, I think we were the first engineering tool or design tool that had an AI chat in it. We shipped that I think a month or two months before GPT-IV became publicly available.”

“Obviously you go to the board and say, hey, I have an idea of taking a billion dollars and giving it to this crazy structure, which we don't even kind of understand. What is it? It's a nonprofit, blah, blah, blah. And saying, go for it. There was a debate. Bil…”

“Because we were so focused on large language models and were researching deeply in this space, we got really early access to GPT-IV. Like summer, 20, 22.”

“My feeling is that you know, GPT-Five got good at verifiable domains. It didn't feel that much better at anything else. The more human angle of it felt like it regressed”

“Go you know, dig up GPT-IV or other models and go to GPT-V. You're not gonna find that much difference of, okay, let's reason together. Let's try to figure out what was the origins of COVID. Because it's still an unanswered question, you know? And I don't see …”

“At that moment if you chopped it down to a bite-sized chunk, like you gave it some amount of context, That a human being, given the same context and the same prompt, would answer in a certain way. He found that he could, ah, you know, given inputs and outputs,…”

“When we trained GPT-IV, we were pretty underwhelmed internally, and then there was a lot of moments, oh, we trained this small, we spent a lot of money on it, and it's kind of like, you know, pretty dumb, at least, like, you know, we have GPT-IV, GPT-III alrea…”

“And if you look back to GPT three, you know, there's a huge leap. I would contend that the leap is similar from GPT four to five.”

“Well, I mean, a big one from just the last few weeks was that we had an IMO gold medal with pure reasoning models with no access to tools from multiple companies. And, you know, that is night and day compared to what GPT-IV could do with math, right?”

“It's like 90 it's like a 95% discount from GPT-IV to GPT-V.”

“It was really not until GPT-IV where these text models started providing real value for people, and we know how to go make the GPT-IV equivalent of video models, and we will do that, and then a lot of these things that are currently annoying, like doors, or, y…”

“Four to four turbo was, like, the model was less than half the size. And four turbo to four O, like four O's cost is way lower than four. And they just kept shrinking the cost.”

“The first one was right when I joined the company in September, 20, 22. We, it was pre-TiGPT. But at the time, GPT-IV already existed internally.”

“We actually did a instruction following post-train on it, so it was really just a data set that was, here's a query, here's what the model completion should be, and I remember that we were like, well, what happens if you just follow up with another query? And …”

“It's an integrated model, so you don't have to, like, pick in our model switcher and know if you should use GPT-IV or O-III or O-IV or any of the complicated things. It's just one thing that, that works”

“Google developed the transformer in 2017. And then they basically let it sit on the shelf, right? Cause it was a research project. They didn't productize it. They were very worried about, you know, from people I've talked to, they were very worried about the, …”

“Treating the model as a product was not a thing before ChatGPT, because we would ship it more like hardware, where, you know, there'd be a release like GPT-III, and then we would start working on GPT-IV, and these weird giant Big spend R&D projects that would …”

“I talked to somebody senior who was there at the time and I asked them, you know, when could you have had chat GPT with GPT four level output if you had just gotten, you know, gone, gone flat out starting in 2017. And they said by 2019.”

“Maybe I'm biased, recency biased, but I think to jump to four to five is most impressive for me, because I guess with 3.5 when we first released it, the most common use case for me then also was still just for coding. And, but now, like, Even though four was b…”

“So using tools, for example, is something that really thinks really important for overall intelligence, GPT two and three couldn't really do that as well. GPT-IV could do it in a more nascent way. And now GPT-V, you get that baked in with the benefit of these …”

“If open models are to win, part of it could be just, like, everybody can have exactly the model they want. We're serving GPT-IV. It's kind of its thing. You can prompt it, but if fine-tuning is more effective than prompting, everybody can have the model. That …”

“Like Olmo-Thirty-Tube is if you squint like original GPT-IV level and fully open.”

“So you go back to GPT-IV, you know, and you're compressing, you know, the entire internet. But now we really don't need to do it because we've trained them to use tools like the internet, right? They're true reasoning engines.”

“In 2023, when we first launched our AI role plays using GPT-IV, back then people were way more concerned about safety, right? And obviously the models now are much better at like refusals and line sharper between what's appropriate and not. But we did see a lo…”

“And I think if you, like, go to another state outside of the Bay Area, probably even in California, outside of the Bay Area, and then you ask somebody how much their life has materially changed, it's, like, pretty close to zero. Real-world inertia is enormous.…”

“The cost of intelligence for a given set of intelligence, let's say GPT-IV, let's say O-one, whatever, it is literally falling a hundred X over the course of one year.”

“Actually for those models, I'd say no need, but if you're using GPT-IV, GPT-IV-O, then it's still worth it.”

“If you look at what it costs to train GBT for originally, I think it was like 20,008, 100 over the course of a hundred days. So I think it costs on the order of like half a million to a hundred million dollars, somewhere in that range. And I think you could tr…”

“I really do think that similar to how we saw with large pre-trained models before with small fine-tuned models made it like, had gains over the sort of GPT-II era, but then were obsoleted by GPT-IV being generally good at everything. I think, to be honest, you…”

“We were one of the launch partners of OpenAI when they first launched GPT-IV”

“Models like O-I are not systematically better than GPT-IV. There's, they're better in certain use cases. Especially ones where you can create data in advance.”

“Garry Tan: He was one of the first people to get access to GPT-IV, and we think of him at YC as the first man on the moon, and that he was the first to successfully commercialize GPT-IV in the legal space.”

“Claude, for instance, the Sonnet 3.5 was very much fire and forget. It would write code in a kind of Pythonic way, just like, let it fail. Don't be careful about it. Whereas GPT four would use defensive programming, use self assertions.”

“Like, the day GPT-IV launched, Morgan Stanley had integrations, because we had been working on it, and these were, like, we had press releases for these, like, we are ready to go”

“In fact, most people using tracks between the world don't even know there's a model called O one or O three and don't even know what the difference is for GPT four.”

“You want to build at the right moment where, like, kind of, like, how we built perplexity around the GPT 3.5 time, not GPT four time. It would have been too late then.”

“I tried GPT-IV just a couple of days ago, and honestly, I wasn't yet as impressed. It just came back with me with too many questions and actually got the implementation wrong too many times.”

“And likewise, when we look at from GPT-IV to DeepSeq VIII it's fallen roughly 600 X in cost. Right. So we're not quite at that 1200 X, but it has fallen 600 X in cost from 60 dollars to less than you know, to about a dollar. Right. Or to less than a dollar. So…”

“For GPT-IV, it was a few hundred million dollars and it's one building full of GPUs, too. GPT-IV 4.5 and the reasoning models, like, oh, one, oh, three were done in a, in three buildings on the same site, and, you know, billions of dollars to, hey, these next …”

“So I expect that to happen for GPT-IV, GPT-IV-O and all of the other models down the road.”

“GPT-IV does not have fluid intelligence, for instance, but O-III does.”