GPT-3

“It's surprisingly good at just freeform, like kind of fluent text generation. So you can tell it to like create a story or create a tweet or create a scientific paper abstract, and it does a pretty good job at that. And before that, whenever I talked to my, yo…”

“We just had a larger data set of, you know, human-like conversations, and we had this you know, very kind of modest size open source model that's only six billion parameters, only trained on less than one terabyte of text. So like, 50 times less data than GPD …”

“The moment something gets released, like, let's say transformers or a particular piece of technology that you think would be interesting or could be worth experimenting with, I think it takes a collective set of months, like, 12, 24 months sometimes to underst…”

“So we built our, the first thing that we called an AI supercomputer. I think we started working on it in 2019 and we deployed it at the end of that year. And it was the computing environment that GPT three was trained on”

“It turns out that Bill Gates, when he saw GPT-III, he's like, oh, this is cool. But we all know GPT-III really didn't have a good handle on knowledge, and he told the OpenAI team, he's like, I'll be impressed if this could pass the AP Biology exam. He literall…”

“I personally, I have to admit, I slept on it. On GPT-III, even saw GPT-III, it feels like, what is this thing useful for? It's like, yes, for marketing, for content writing, for creative first draft. Didn't really click for me.”

“Well, I went through this myself watching the technology that we built to do document extraction at Impura become, you know, totally irrelevant.”

“Anthropic basically didn't exist five years ago. OpenAI was still quite early. I think GPT-III just came out and SpaceX was trading at 80, a hundred, something like that.”