GPT 3

“It feels like, to give an analogy, it feels like we are in the kind of GPT-III era for robotics models you know, in terms of the development of language models, right? So, like, we're starting to have models like Gen-one That feel like they are getting to brea…”

“Anthropic basically didn't exist five years ago. OpenAI was still quite early. I think GPT-III just came out and SpaceX was trading at 80, a hundred, something like that.”

“The only commercial use case that was working was copywriting, you know, so you pay like some marketing firm, 20 bucks and they paid us 20 cents for the AI to like write a landing page or whatever.”

“OpenAI came to us when they were training GPT-III and they wanted to teach GPT-III to code. So we collaborated with them on teaching the models to code and to do tool use, function calling”

“On 86 out of a hundred of those questions, three out of three attorneys said yes, and that was the Oh my God moment for me and for my co-founder, or so we need to do something in this industry.”

“Like I saw some people doing demos, but like in like a CLI or something like that, but there was no product doing like this model, but with additional data on top of it.”

“It was medical spam, like advertising different drugs to people, right? It's like not something we ever would have thought of as a problem.”

“So basal alarms were scoring extremely low on V-one, like sub-ten percent, basically. And, I mean, it was true of the original, like, GPT-III actually scoring zero, but that's even true of the latest basal alarms today, you know, as of March.”

“And I got GPD three to do in context learning, few short learning, and you know, it was kind of the First, at least to me, it was the first known implementation of RAG, Retrieval Augmented Generation, which I used to solve this problem of querying, getting GPT…”

“I primarily called email Greg Brockman, who's a co-founder and president, and I cc'd Sam Altman, and I got a reply from Greg in less than an hour from sending an email when they probably had less than 50 users for GBT-free.”

“The way we got to it was we, in early, we started experimenting with using GPT-III to generate answers. For Quora. And we compared them to the human answers and sort of realized that they weren't as good, but what was really unique was that you could instantly…”

“When we trained GPT-IV, we were pretty underwhelmed internally, and then there was a lot of moments, oh, we trained this small, we spent a lot of money on it, and it's kind of like, you know, pretty dumb, at least, like, you know, we have GPT-IV, GPT-III alrea…”

“And if you look back to GPT three, you know, there's a huge leap. I would contend that the leap is similar from GPT four to five.”

“If you just take just the old first generation GPT-III model, which had a context window of 2000 tokens and a vocabulary of 50,000 next tokens or 50,000 tokens, then the size of it, the number of rows in this matrix is more than the number of atoms across all …”

“Like the public estimates for GP three, I remember, were that it cost five million dollars to train, which you're like, on the one hand, five million is kind of a lot, but it's like a lot for an individual person. It's not really a lot from like a company pers…”

“Obviously, the cost to serve a model quality of GPT-III has tanked, you know, like... Yeah, it's like 2000 times cheaper now.”

“Treating the model as a product was not a thing before ChatGPT, because we would ship it more like hardware, where, you know, there'd be a release like GPT-III, and then we would start working on GPT-IV, and these weird giant Big spend R&D projects that would …”

“Actually, the original reason for building GPT-II and GPT-III, it was an outgrowth of the kind of AI alignment work that we were doing, right? Where myself and Paul Cristiano and some of the Anthropic co-founders had invented this technique called RL from huma…”

“In five years we have gone from this, like, thing that could barely write a sentence to a thing that is like, you know, PhD level intelligence in most areas.”

“But it rumored that to go from GPT-III, which, you know, had existed for a while and didn't You know, impressed some people, but was certainly not the breakout moment ChatGPT was. To ChatGPT, it was like a one percent increase in the training costs.”

“If we had, you know, we got lucky with timing. If we had founded the company six months before we did, GPT-III would not have been good enough, the equivalent of it, at that point in time, and we wouldn't have been able to create that initial magical experienc…”

“When I started investing in generative AI all these early foundation model things, et cetera, basically nobody was doing it. And it was all out in the open, right? GPT-III had just dropped. It was clearly a big step function from two. If you just extrapolated …”

“So when we looked at GPT-III, the cost fell of 1200 X from GPT-III's initial cost to what you can get LLAMA three point two three B today, right?”

“I can see now, having been through this cycle a few times, that we're nearly there with memory, personalization, and actions. It's really at the GBT-III stage, so it's really buggy and stuff, but when it works, it's breathtaking.”

“I mean, there are folks who have trained models that perform as well as GPT-III that are a hundred times more inference efficient that cost an order of magnitude less to train and yet they can still deliver the same predictive capability.”

“OpenAI even had a bot when I worked there called the Truthbot, which John, John built with this team. Where you could ask it a question and it'll go and search the web, and then it'll give you an answer with some sources. And it was very slow, and it was built…”

“When GPT-III came out, I think we were one of the first companies to build anything on top of it. The first thing we built was like, you highlight a piece of code and you explain it.”

“It was really hard to build anything like that with GPT-III. It was expensive. It was slow.”

“When you 10 X the energy and tokens and data and all the work on the training, you do get a smarter model, but the increases have been diminishing. So as you spend 10 times more, It, you don't get the same increase from GPT three to GPT four.”

“O-one is actually the first model where I'm getting more impressed by it the more I use it. So, like, when ChatGPT first came out, right, I think it was GPT-III. And at first it seemed like, oh wow, this is amazing, it can actually create text that sounds like…”

“So we, we've always had our own technology down to the LLM. There was a very brief period, literally like three months, where we, for some of our apps, used GPT-III because it was just so much better.”

“Most of the compute was going to a product called Nest, which was basically GPT-free.”

“But, with the possible exception of copywriting, no great businesses were built on GPT-III.”

“Well, I went through this myself watching the technology that we built to do document extraction at Impura become, you know, totally irrelevant.”

“I spoke to a very large financial institution, spent a huge amount of money building a GPT three powered sales assistant tool that as soon as chat GVD came out was instantly obsolete.”

“I think for something like a GPD three level model, it's a 10,000 X speed up in efficiency while losing nearly not much at all in accuracy.”

“ChatGPT was really just GPT-III with instruction. It was basically like more chat tuning, but GPT-III API came out, I think, over a year before ChatGPT did, but only developers could play with it.”

“There's an example of an AI-native companionship product where they actually did this test where they used GPT-one, GPT-two, GPT-three, and they actually tracked the cohort of user retention using each of the models. And it's very clear as the model quality im…”

“We actually tried to lowball the system in that we published a research paper called like language models are few shot learners. I don't think we even tweeted about it. We tried to like public publish it publicly, but also be like very quiet and see, see how q…”

“We actually were using Transformers at Stitch Fix, like, before the GPT-III model, so we were just using Transformers for recommendation systems.”

“I think GPT-III was a big change because it kind of said, oh, now is the time to build to you that we can use AI to build these tools. And then GPT-IV was maybe a little bit more of an extension of GPT-III. It felt less like a level, GPT-III over GPT-II was li…”

“I personally, I have to admit, I slept on it. On GPT-III, even saw GPT-III, it feels like, what is this thing useful for? It's like, yes, for marketing, for content writing, for creative first draft. Didn't really click for me.”

“So I think that we continue to feel like graphics and these foundation models for anything really related to pixels, but also definitely images continues to be very under invested. It feels a little like graphics is in like this GPT two moment, right? Like eve…”

“There was also a misconception on GPT-free and basically in 20, 21, every paper made this mistake.”

“And all of a sudden in with GPT-free, this tide kind of reversed and companies started to be more opaque about what they were doing because they realized there was actually a very big market. And all of a sudden in 20, 22, on the important aspects of AI and on…”

“Fine tuning is the technology that got from a research project in 2020 called GPT-III and turned that into ChatGPT, a billion dollar app, right?”

“I think that GPT-IV, my hypothesis is that the jump from four to 4.5, or four to five, will be smaller than the jump from Three to four.”

“And I think OpenAI did an excellent job with respect to that. Like, they released GPT-III, and they were like, we are just going to give access to researchers who we are going to vet. That's an amazing way to give access. Now it's kind of pretty standard way, …”

“When we looked through it was like, you know, it's hard to estimate, but it was something like, you know, maybe .1% to one percent of the data that we scraped was Python data.”

“GPT-I had roughly a hundred and twenty million parameters that it was trained on. GPT-II had 1.5 billion. GPT-III had a hundred and seventy five billion, and GPT-IV, OpenAI hasn't announced, but it's rumored that it has about 1.7 trillion parameters that it wa…”

“It turns out that Bill Gates, when he saw GPT-III, he's like, oh, this is cool. But we all know GPT-III really didn't have a good handle on knowledge, and he told the OpenAI team, he's like, I'll be impressed if this could pass the AP Biology exam. He literall…”

“Three weeks ago, there was a paper about Lima. So, so this Lima paper shows that with only 1000 question and answer examples, so very, very small data sets they get something for, use for fine tuning, so the second stage, they get something that performs bette…”

“So we built our, the first thing that we called an AI supercomputer. I think we started working on it in 2019 and we deployed it at the end of that year. And it was the computing environment that GPT three was trained on”

“The tool that we had was good for 2016 through 20 19 deep learning, but as of GPT three deep learning, it wasn't good enough, and so we had to upgrade to this kind of new paradigm that we introduced.”

“The moment something gets released, like, let's say transformers or a particular piece of technology that you think would be interesting or could be worth experimenting with, I think it takes a collective set of months, like, 12, 24 months sometimes to underst…”

“We just had a larger data set of, you know, human-like conversations, and we had this you know, very kind of modest size open source model that's only six billion parameters, only trained on less than one terabyte of text. So like, 50 times less data than GPD …”

“It's surprisingly good at just freeform, like kind of fluent text generation. So you can tell it to like create a story or create a tweet or create a scientific paper abstract, and it does a pretty good job at that. And before that, whenever I talked to my, yo…”

“Every venture capitalist I met has already had a multi-day offsite to stop and reflect about what its ramifications can mean in the medium term and what it may mean, not only for their portfolios, but their theses.”

“If I hadn't been on Twitter six months before we launched Jasper, I wouldn't have seen GPT three come out. I wouldn't have seen some of the cool demos and I wouldn't have thought, well, I think we could do that for marketing content.”

“GPT three and now four is, is a reasoning engine. And Sam Altman has talked about this. It's not a knowledge base where it's like, and so people kind of latch on to this fact that, oh, the data that it has is from September, 20, 21, then I'm going to teach you…”