The Exchanges

Q Great. All right, Nathan, how about you? So you, you're, you're a man of, uh, many talents. You do research. You write, uh, this, uh, very interesting, um, uh, blog slash newsletter called Interconnects. Uh, you do podcasts. You do, like, a bunch of different things. So tell us about your journey.

A Yeah, I say I wear many hats to try to get the things that I want to do done. Um, I showed up to Berkeley as a EE mostly PhD admit in 2017, and then I saw that AI was happening, and I decided that I want to try to do this, which started by going to all the names that people know, like Sergey Levin and Peter Abil, and asking to be in their group, and then they respectfully say no. And then starts the long process of becoming, um, learning how to actually do it without being directly embedded in these elite groups, which was a mix of robotics and reinforcement learning and finding my way there. So my PhD was in mostly model-based reinforcement learning, and then my one research avenue job was to go join Hugging Face when they said they were going to make an open source version of DeepMind and do a bunch of research. Realistically, my job was not that impactful or useful at Hugging Face until ChatGPT came out, and then I was like, oh, I should maybe just learn about RLHF, and that got very immediate traction as somebody trying to work in public with the team there, so. Like Lewis Tunstall and other people at Hugging Face are still doing a great job on this, and we worked together for a while, and then mostly I was just like getting burnt out on remote work and met Luca in Hawaii at a fun conference. It was like, wow, I could have real life friends, and I joined the AIQ to work in …

AI assessment note: “I showed up to Berkeley as a EE mostly PhD admit in 2017”

Q podcast, uh, with Richard Sutton that was, uh, Effectively saying that, like, RL was the way to go and that pre-training and LLMs was a little bit of a flawed premise because it was sort of an imitation of, uh, reality basically doing the way humans described, uh, reality as opposed to being confronted with the actual reality through RL. Do, do you guys have any quick take on that?

A My take is that a lot of people are being exposed to Rich Sutton for the first time, and Rich is a font of wonderful ideas, but Often not ones that are going to be immediately practical. This is how you get things like creating reinforcement learning, but not necessarily things that are going to impact what GPT six is. So I've been on a critiquing rich life for many years before this in terms of making people try to interpret his ideas as realistic. I think the one from 2021 or 2022 is it's like reward is enough paper, which essentially is an argument that A reward function is sufficient to get any intelligent agent that you want. So I think that that's actually, like, rather than the technical debate as an entertainment of the whole community being nerd sniped for the first time by that, it's a distraction.

AI assessment note: “Rich is a font of wonderful ideas, but Often not ones that are going to be immediately practical.”

Q You want to give the, the Plain English definition of RLVR versus RLHF?

A RLVR and verifiable rewards is in the name. I think essentially the reward that you get from the quote unquote environment, which is like the completion or the greater, is whether or not you got the problem right. RLHF, the reward is some, is a essentially a reward model, which is rating the quality of the response based on a proxy to what humans would like. So It's described as being a much like the RLVR reward is much easier to understand because these reward models tend to have a lot of problems and you can over optimize them much more easily because the reward models will pick up on features that are maybe emojis or something like this that you don't actually care about where RLVR is much better matched to like performance characteristics rather than style.

AI assessment note: “the reward that you get from the quote unquote environment... is whether or not you got the problem right”

Q And then why is that? So you, you, you have a pre-trained model, but you, um, uh, for supervised fine tuning, You did still using a different model. Why is that in, in simple terms?

A Essentially because our small model is not going to be able to output as strong of text, even if, so there's a kind of a fork in process where I'm talking about a small model and if we had a bigger model, what we would do is do a lot of reinforcement learning to start. And the model then would take time to learn these interesting behaviors and have strong performance. But with a smaller model, the ceiling on that is fairly low. Like it just doesn't have the capacity to learn from these harder math problems. So what the common practice is, is you take the absolute best reasoning models you can get that are openly available with a good license where you can just generate new data yourself and train on it and release it to the community, which is something we've been seeing a lot of this year. And then therefore the models that are from closest to the frontier in performance with good license all happened to be Chinese models throughout the year for, for this case. And I think in our case, even if GPT OSS had existed, I don't think we would have used it. For synthetic data in this, because that model is really designed for, for tool use, which is something that we did a bit of in this project, but not in the sense that, um, that model is, which is like this many hop agentic reasoning with search and stuff. So like deep, the deep seeks and quen's of the world are just powerhouses a…

AI assessment note: “our small model is not going to be able to output as strong of text”

Q All right, so let's talk about the next, uh, stage in the pipeline, stage five, uh, DPO and preference tuning. What, what is that? What does that do?

A Yeah, so this is one of the things that is Um, thought of as like, hey, let's try this. We're not sure if it'll work kind of later in the process when you spend a lot of time on other things and it works very well. I think, um, DPO or direct preference optimization is not exactly new. I think it's a, it's a way of optimizing for preferences. It's related to this whole RLHF thing that we mentioned. Technically speaking in one sentence, it's a analytically derived, um, Loss function that, like, is essentially applying stochastic gradient descent to the RLHF objective. So it becomes much easier to implement than other things. And we use this in the past with OMO two, with two-lute three, two-lute two, other OMOs. And the question was, like, can we apply this out of the box on top of a reasoning model? And we knew that it works in many different situations because we weren't sure what would happen with these long reasoning traces being included in the loss function and so on. So then, essentially, like, there's a student, Scott, that has been working on this, what he calls the Delta Learning Hypothesis, which is like a intuition for understanding DPO as being more about the contrast between your chosen and rejected examples. So the core of preference learning is that you have kind of pairs or some grouping of completions to the same prompt. So you have one question with multiple co…

AI assessment note: “it's a analytically derived, um, Loss function that, like, is essentially applying stochastic gradient descent”

Q And then Quen is, is widely, widely used in a way that people may not have completely realized, right? There was a, as an anecdote to the, the example of Airbnb talking about using Quen over ChatGPT, if you Weeks ago, but do you have any sort of stats or anecdotal evidence on, on the usage of Quinn?

A The other famous quote was a Martin Cassato quote in The Economist, where he said, 80% of companies are building on Quinn. That has been corrected, whereas 80% of companies building with open models are using Quinn, which is like 16 to 24% of his portfolio, which is still a lot. Like, it's a meaningful amount of people are trying open models for things, and most of them are using Quinn. And then there's the likes of, like, Cursor released their own model, like Composer too. It's accepted that it is built on a large Chinese MOE, whether of some sort that was released openly. Um, there's some obvious tells of like, it's switching to Chinese and things like this, but that's just like, that is the sort of company that doesn't want to pre-train their own models, but has immense value in specifying models for their use case. That is just going to build on these, on these great models. And I think they would want more options to choose from as they try to sell into more markets, I think. Realistically, it's a thing where a lot of US companies don't want to deploy Chinese models, I think. Currently, a lot of the stated reasons are just unknown unknowns and things you can't prove. Like, you can't prove that the models aren't doing certain backdoors, where I'm fairly certain they definitely aren't now. But like, just because you can't prove it, it makes this kind of weird market dance, w…

AI assessment note: “80% of companies building with open models are using Quinn”

Q Congratulations. Best form of flattery. All right. Switching tags and, um, in an effort to, to, to make those conversations, uh, educational for a broad group of people. Uh, so one of the key aspects of the release is the thinking model. Could you remind folks, uh, what a thinking model actually is versus, uh, other form of soft models or prior generations of models?

A A lot of people have heard about inference time scaling, which makes sense, which if you spend more compute at inference time, you get a better answer. A thinking model is really a way that Train the model to exploit that a lot. So you spend a lot of tokens, which is the tokens are usually hidden from the user as like a long chain of thought. And the model therefore kind of has a step change where it's way better at math tasks, coding tasks, agentic tasks. I think we, we have some, like our future plans are adding more tool use to the model. So we're not talking a lot about like agentic search or agent code execution on the fly and stuff for this model, but like, Building thinking models is the gateway to doing a lot more interesting things like cloud code, like maybe we'll have all no code next year and all these things that we want to do, like, like the thinking model has just been the thing in 20, 25 that use a lot more compute per answer model gets way better at various things.

AI assessment note: “A thinking model is really a way that Train the model to exploit that a lot.”

Q So to play it back, you're both saying, uh, yes to AGI, but no to discontinuity slash singularity. And one, is it fair? And, and, and two, if that's, um, what you're saying, then Uh, for AGI using the current paradigm, basically what we just described in the last hour of pre-training plus RL gets us there.

A I think the AGI word is actually pretty not useful. I think that how I describe it is that big tech has all collectively realized that these language models plus scaffolding is going to unlock absolutely incredible value. And I have very high probability, barring extreme geopolitical situations, that big tech Executes on this vision across the two to five years to build. 95 to 98% of the way there of what you can do with our physical power constraints and what an LLM's ability is. And I think that that will be extreme. Like the transformation from that by 2030 is going to be so powerful across society. There's a bunch of long tail, like there's going to be mass societal readjustment to what the Internet and media and information means and. Like, within five years, and that's mostly why I do this, and I think, like, whether, debating whether or not it's AGI is kind of secondary to the fact that this is coming, and we want people to study and understand what is happening.

AI assessment note: “I think the AGI word is actually pretty not useful.”

Q All right, and to complete the journey since we started talking about RL, so RL VR, Reinforcement Learning with Verifiable Rewards, let's spend a little bit of time on that sixth stage. In particular, Nathan, I understand that's your baby, or you're one of the fathers of the baby. Do you want to walk us maybe a little bit through the History quickly?

A I mean, I think that I'm the person that got to bring it publicly to the world. It's well known that people across the industry have been doing this for years, and then the techniques started to get far more impactful. It's, it's broadly taking existing reinforcement learning algorithms or downstream evolution of, uh, like, proximal policy optimization, PPO, which is like an evolution of reinforce, and then DeepSeq had their group, group relative policy optimization. I always try to say, like, group robust, and it's, I think it's group relative. Um, and like all these algorithms are really quite similar, and it's, you're, you're training the models with whether or not they got the answers right, or in the case of code, whether or not the tests execute and don't fail. I think one of the famous examples is that doing too much of this kind of, or racing to get the low hanging fruit from this RL approach is what makes all these code models do all these try except things to avoid errors because they accept all the errors. I think that is just because The gains that you get in the model being useful is so much higher than the annoyance and the fact that it also does these stupid things and will fix the stupid things eventually. In the case of this ULMA model, it's not anything crazy. We have, we take a very, we cast a wide net on RL math problems. We do some data comparisons to see w…

AI assessment note: “I think that I'm the person that got to bring it publicly to the world.”

Q to just a broad group of people who are curious to understand how AI works. So, um, SFT is not RL yet, right? That's a supervised fine tuning. So, uh, that means that you basically show The model, some like a golden, like a gold copy of what good looks like, and you train it based on that label data. Is that, is that a good way to describe it?

A Yeah, so it's the same loss function as pre-training, which is you're predicting the next token. In this case, what it looks like is a question could be like, I don't know, like an Amy style, like a really hard math question. It would be like, list all the prime numbers within some constraint of x and k. And it's like this one sentence that is really hard, and then the model generates 30,000 tokens of let me think about this and do this and to test this, I'll have to use this theorem and hypothesis, which is like the third, we were talking about token intuitions for a bit, but 30,000 into tokens to solve a math problem is pretty mind bending. So if I were to sit there and read this, it would be hours of me just trying to read this one math solution. So these models are very, um, Unintelligible in many ways. I think the reasoning model sometimes will go into like a bout of guess and check for hundreds of attempts before realizing that they can no longer guess and check. I mean, like this is like our reasoning model. It's a, I think the frontier models could have probably done this and fix this issue, but they're just really, really, really odd things in these tokens. But even with that, doing this next token prediction is An incredible foundation of performance that many people use. So it's like, it's, it's not matching any sort of human reasoning or things that people might wan…

AI assessment note: “Yeah, so it's the same loss function as pre-training, which is you're predicting the next token.”

Q And is there more of, uh, an emerging organized response in the U.S.? Uh, I know you're involved, or perhaps behind the, the Atom. Project.

A I think any concerted response, you only see when it actually is public. And I think there's a lot of investment at different stakeholders and conversations that are happening, but like that's not that useful. So it's like, I don't, I don't have the proof for you, but I do think the right people are talking about it and want to invest more. Cause realistically the cost is not that high relative to the trillion dollar build out of AI infrastructure. It's like, oh, if point oh, one percent gets a better, great open models, like we should probably do that. I think that's actually not that complicated. It's just like, how do you get the a hundred million dollar line item to the right people that have the talent to Do it like, oh, okay. The right incentives. It's just like, oh, okay. It's a lot like the reflection news is, is like, okay, that's, that's probably a good solution for a couple of years. Like they have enough money and they have a strong base of talent. And it's like, okay, that's, that's like a major checkbox. We need to have some, we need to have diversity there. Cause the llama thing could happen again or it goes away, but like, okay, like looks like a small snowball, but hopefully grows in the coming months.

AI assessment note: “I don't have the proof for you, but I do think the right people are talking”

Q So in other words, did they, did they teach to the test versus, uh, enabling true thinking?

A Yeah, so I don't think, like, Quen didn't, it's a, it's a gray zone. Like, I mean, it's not, I think all the Frontier Labs will do this to some extent, which is how their task is. You have a team member that's tasked with, um, improving an evaluation. And then the easiest way to do this is to train on test. But they all have dignity as, like, elite scientists where they won't do this. And the next closest thing is you do some sort of paraphrasing of the test set to Create true data, new trading data. So therefore you're not technically cheating, but you're potentially like, it's like where in the spectrum of you scrape GitHub for math problems versus you paraphrase the evaluation set, like where do you draw the line on like actually calling it cheating? And, um, different people have different answers, but mostly like, I think a goal that we kept coming back to, because we understand that Olmo is not like Like you can look at the numbers. We're getting close to Quinn three with reasoning or without, but like this is not a six hundred billion parameter model that people are going to immediately download and run all no code on or anything. But we want to make sure that our core audience could do the research that, that we want to do with, with confidence and debate it. So we want to give people access to every stage and you can then see how this impacts this new important area of…

AI assessment note: “it's a gray zone. Like, I mean, it's not, I think all the Frontier Labs”