GPT-3

“While investigating AI bias, it gave some odd answers that I wasn't expecting. Things that GPT-III would never generate. Things like, hey man, I'm just trying my best. These are really hard topics to talk about. Can we talk about something else? And I have nev…”

“And I think OpenAI did an excellent job with respect to that. Like, they released GPT-III, and they were like, we are just going to give access to researchers who we are going to vet. That's an amazing way to give access. Now it's kind of pretty standard way, …”

“We actually tried to lowball the system in that we published a research paper called like language models are few shot learners. I don't think we even tweeted about it. We tried to like public publish it publicly, but also be like very quiet and see, see how q…”

“I can see now, having been through this cycle a few times, that we're nearly there with memory, personalization, and actions. It's really at the GBT-III stage, so it's really buggy and stuff, but when it works, it's breathtaking.”

“I mean, there are folks who have trained models that perform as well as GPT-III that are a hundred times more inference efficient that cost an order of magnitude less to train and yet they can still deliver the same predictive capability.”

“So when we looked at GPT-III, the cost fell of 1200 X from GPT-III's initial cost to what you can get LLAMA three point two three B today, right?”

“Actually, the original reason for building GPT-II and GPT-III, it was an outgrowth of the kind of AI alignment work that we were doing, right? Where myself and Paul Cristiano and some of the Anthropic co-founders had invented this technique called RL from huma…”