The Exchanges

Q like, diversity and generalization across the different things you can be doing, but it, it's a, it seems like a very large space and scaling RL, like, new gen RL is not, it's just not obvious, like, how, to me, it's not obvious how you do it or how you choose the path. Is there some sort of organizing framework that, You know, you guys have that you can share?

A I mean, I, I don't know if there's, like, one organizing framework. I think there are a few, like, factors at least that I think about in, like, the very, very grand scheme of things is, like, how much, like, in order to solve this task, like, how much uncertainty with the environment do I have to, like, wrestle with? Like, um, for some things where it's, like, this is a purely fat, like, who was the first president of the United States? Like, there's zero, like, Environment I need to interact with to, like, reach the answer to this question correctly. I just need to remember the answer and say the answer. You know, if I want you to, like, write some code, you know, that, like, solve some problem. Well, now I have to deal with a little bit of, like, not purely internal model stuff, but also, like, okay, I need to execute the code and, like, that code execution environment is maybe more complicated than my model can memorize internally. So I have to do like a little bit of like writing code and then executing it and making sure it does what I thought it did and then testing it and then giving it to the user and things get like the amount of that sort of stuff outside the model that you have to like, you know, you can't just recall the answer and give it to the user. You have to like test something and, you know, run an experiment in the world and then wait for the results of tha…

AI assessment note: “there are a few, like, factors at least that I think about”

Q does too, uh, a company that wants to generate data for the labs in some way. And, um, or it's, you know, access to human experts or whatever it is, but, like, you know, there's, there's infinite variations of this. Um, uh, if you could wave a magic wand and have, like, a perfect set of data, like, what would it be? That you know would advance model quality today?

A This is a dodge, but like uncontaminated evals, um, always super valuable and that's data. Um, and I mean, yeah, like you want, you know, good data to train on and that's of course valuable for making the model better, but I think it is often neglected how also important it is to have high quality data, which is like a different definition of high quality when it comes to an eval. Um, But yeah, the eval side is like often just as important because you don't, you need to measure stuff. And like, as you know, from, you know, trying to hire people or whatever, like evaluating the capabilities of like a general, like capable agent is really hard to do in like a rigorous, you know, way.

AI assessment note: “This is a dodge, but like uncontaminated evals, um, always super valuable”

Q certain basic tasks done, and then there's another model which you, Upload a legal M&A folder, and it takes a day to think, and it's slow and expensive, but then it produces, you know, output that would take you a team of people. You know, a month to produce, or how do you think about the world in terms of how, how all this is evolving or where it's heading?

A You know, I, I think for us, like unification of our models is something that, you know, Sam has talked about, uh, publicly that, you know, we have this big crazy model switcher in ChatGPT and there are a lot of choices and, um, you know, we have, uh, a model that might be good at any particular thing, you know, that a user might want to do, but, That's not that helpful if it's not easy for the user to figure out, well, which model should I use for that task? Um, and so, yeah, making the models better able, you know, making this experience more intuitive is definitely something that is, is like valuable and, and something we're interested in doing. And that, that applies to this, you know, uh, question of like, uh, you know, are we gonna have like two models that, you know, people pick between or a zillion models that people pick between or, Do we put that decision, you know, inside the model? Um, I, you know, I think everyone is gonna try stuff and figure out what works well for like the problems they're interested in and like the users that they, they have. But, um, but yeah, I mean, that, that question of like, how do you, you know, make that, uh, that sort of decision be like as, you know, effective, accurate, like intuitive as possible is, is definitely top of mind.

AI assessment note: “unification of our models is something that, you know, Sam has talked about”

Q specifically for it to be better at that, or other things that you'd built against it, or how did you think about the data training for it, the data that was used for training it? Like, I'm just sort of curious, like, how that product, if it all is a branch off of, Off of this and how you thought about building that specifically as part of this broader effort.

A I think when we, uh, think about, like, tool use, I think browsing is one of the most, like, natural places where, you know, you, you think of as a starting point of, like, okay, like, and it's, it's not always easy. I mean, like, the, you know, initial kind of, uh, browsing that we, uh, included in GPT-IV a few years back, like, it was hard to make it, you know, work in a way that felt, like, reliable and, like, useful. Um, but, you know, in the sort of, you know, modern, these days, Last year, you know, uh, two years ago is ancient history. Um, I think it feels like a natural place to start because it's like so widely applicable to so many types of queries, like anything that is, you know, requires up-to-date information, like it should help to browse for. And so, um, in terms of like a test bed for, Hey, like, does, you know, the way we're doing RL, like, does it really work? Or, you know, can we really get the model to learn, like, Uh, longer time horizon, kind of meaningful extended behaviors, like, it, it feels like kind of a natural place to start in some ways, in that it, you know, also is fairly likely to be, like, useful in, in a relatively short amount of time, so it's like, yeah, let's, let's try that. Um, I mean, you know, in, in RL, like, at the end of the day, you're defining an objective, and, uh, if you have an idea for, like, Who is gonna find this most useful…

AI assessment note: “in terms of like a test bed for, Hey, like, does, you know, the way we're doing RL”