GPT 2

“In 2019, 20, 20 time GPT-II had just been trained. And that was, I think, one of the first language models that came together and was coherent over longer horizons. And there was a variant of GPT-II that that was called Codex. And this is kind of the same name…”

“And you can actually, in fact, Take a GPT one or two model and with a few, I mean, few lines of code almost, you can transform it into the latest let's say deep seek version, 3.2 architecture. It's not like a big leap. It's still the same as scaffold.”

“Yeah, like I would say that robotics is in kind of like the GPT-one to GPT-two area right now.”

“We're in like the, you know, the GPT-II era of memory”

“I find it helpful to frame these in terms of like GPT one, two, three, four, five capabilities, right? And I think most people would agree we're somewhere between GPT one and two, right?”

“I'd say that maybe the neural net we produced, you know, it's a 40 B neural net trained on, you know, like 13 trillion base pairs or something like that. The results to be felt like GPT one, maybe starting to be GPT two level, right? It's like accessible or, a…”

“Actually, the original reason for building GPT-II and GPT-III, it was an outgrowth of the kind of AI alignment work that we were doing, right? Where myself and Paul Cristiano and some of the Anthropic co-founders had invented this technique called RL from huma…”

“If you try to do the reasoning paradigm on top of GPT-II, I don't think it would have gotten you almost anything.”

“We actually helped OpenAI with GPT too.”

“Scale partnered with open AI very, very early on before chat GPT came out. This was like very early on RLHF when they were trying to tune models to summarize better based off of Reddit passages. And this is on GPT two.”

“Because between, like, GPT-II and GPT-IV. There's been this 50,000 X scale up of base models that has resulted in, in, in basically a flat curve. On Arc.”

“Yeah, it's kind of like GPT-II for that approach, and the idea is that that could scale further over time, and it's kind of the initial early proof point that you feel like something really real is happening. And then it's just going to be subject to the same …”

“You can train it on a single node of H-one-hundreds in about 24 hours, and that costs roughly 600 dollars.”

“At the time of that post, we were using, in LL and that's in 30% less memory, and we were 20% faster in training, just the truth.”

“And Andre Karpathy, who was a co-founder of OpenAI, who has left, actually now has an open source version of GPT-II where you can train your own GPT-II from scratch for less than 100 dollars in compute costs.”

“I think within five years, you're going to have models of this size on your phone.”

“When we were training GPT-II, we trained GPT-II in various different sizes. And at the smallest size, the model was just, like, unable to do three-digit arithmetic. But as the models got bigger and bigger and bigger, we didn't change anything else. We just had…”

“There's an example of an AI-native companionship product where they actually did this test where they used GPT-one, GPT-two, GPT-three, and they actually tracked the cohort of user retention using each of the models. And it's very clear as the model quality im…”

“We do also use, like, T-Five-based models, even, even now but started, yeah, started with GPT-II.”

“So I think that we continue to feel like graphics and these foundation models for anything really related to pixels, but also definitely images continues to be very under invested. It feels a little like graphics is in like this GPT two moment, right? Like eve…”

“Even kind of prior to this kind of latest big shift in the AI landscape, you know, we were using GPT two as part of us, the square messages product being you know, a virtual assistant to help customers, you know, answer responses to customer inquiries and vari…”

“GPT-I had roughly a hundred and twenty million parameters that it was trained on. GPT-II had 1.5 billion. GPT-III had a hundred and seventy five billion, and GPT-IV, OpenAI hasn't announced, but it's rumored that it has about 1.7 trillion parameters that it wa…”