The Exchanges

Q So what do models need to, um, know about people? Or, like, what capabilities are they, um, either missing or have not been elicited from them?

A The most fundamental thing. Is that the models kind of don't understand the long-term implications of the things that they do and say. When you treat every turn of a conversation as kind of its own game, and you, you know, you basically think of it as like, okay, you had this interaction. You're done. You need to make sure that this one response has all of the possible answers, has all of the possible content. You don't ever, like, ask questions. You don't ever, like, try to clarify things. You don't really tend to express uncertainty. Um, you don't tend to be proactive. You don't tend to think about the long-term, uh, like, like, you see a lot of, like, even single-term side effects of this kind of regime. Like, and most of them are treated as kind of their own problems to solve. You see issues around, like, that, that people highlight around, like, sycophancy. You see issues that, you know, that there was recent news around, like, you know, the psychosis stuff. There, there's a lot of these, like, uh, harmful effects that you, Get if you think about things in this very single task or like task centric way. Um, but if you have models that actually consider, you know, the long term implications of, oh, hey, if I tell this person to start like a, you know, a company that, you know, sells gloves for catching ice cream. If I like tell them that that sounds like a good business ide…

AI assessment note: “models kind of don't understand the long-term implications of the things that they do”

Q Can you give a broad, like, we have, um, everything from researcher audience to business person audience here. Can you give a broad intuition for a star?

A I guess the intuition is, if you have a model, and it's able to solve these, like, basic, like, these, like, slightly harder questions by thinking about them, then what if you actually teach it? Like, hey, this solution that you came up with, that got you to the right answer. Good job. Or, you know, if you Or if the model didn't, then you basically, like, don't reward it. I guess the original version of SART actually had, like, or yeah, there were, like, no, there wasn't a baseline at the time. We compared it to Reinforce, which is this, like, popular algorithm in, I guess, reinforcement learning. Like, very simple, like, policy gradient thing. But yeah, I guess, you know, at the time, it was, like, a very simple algorithm. Just, you know, you Uh, iteratively generate solutions. If the solutions get you to the right answer, you learn from them. If they don't, you don't. And then you just kind of keep doing this as the model solves harder and harder problems, and then learns from harder and harder problems.

AI assessment note: “iteratively generate solutions. If the solutions get you to the right answer, you learn from them.”

Q self-consistency issues between, like, I want to learn this today, but I don't actually want to do the work. I want to eat cake, but I want to be in shape as well. Like, you know, we have different time skills and change our minds. I'm just constant distribution shift, like, and then you can't possibly bring all of us under distribution. Like what, how do you react to that?

A I think to a certain extent, it's probably a little bit true. It's not easy to, to, to build these like really good models of people. But I do think that the task for the model needs to be that it should be trying to do that. Like the model needs to actually be like trying to learn all of these, like trying to learn about you, trying to learn about, you know, the things that you care about. Like the, the actual objective of the model needs to be, to kind of understand you. And like, it probably won't be perfect. Like, but boy, you know, like, you can be a lot better than the current models. Like, uh.

AI assessment note: “I think to a certain extent, it's probably a little bit true.”

Q What, um, do you think is the most dominant explanation for attempts to use models in very, very, more verifiable domains like code still failing at sophisticated tasks? Is it just like the wrong context has been fed to them? Is it, um, context window is simply not large enough to support the, like, scratch pad and continual testing? Like, what, why, in those domains, what is the biggest challenge?

A Part of it is there's, I think, a balance. When people kind of want to give users these models, it's actually important that they're not annoyingly slow. And so I think there's actually like a number of problems where like, if you gave the models more time, you know, they would actually be able to answer better. But for example, in the kind of coding context, you kind of have to be reasonably responsive. At least it depends, it depends on the kind of setup, right? Like, if you look at products like, you know, OpenAI's Codex, um, which, you know, is kind of this longer-running background thing, uh, versus, like, uh, Cursor, which is, like, you know, more interactive. You, you have a bit more luxury with those, um, more background approaches, uh, to tackle harder problems, I'd say. Yeah, I think, I think it's a, it's a tricky question. A lot of things depend on how far the distribution of What you're, what you're asking is from the distribution that the models were actually trained with. Um, so, you know, if you happen to be asking a problem that's very similar to the kind of problems that it's seen before, then, you know, it'll do great. Uh, and if you're asking a problem that's, like, very, yeah, out of domain. It, so, like, to some extent, this question is kind of hard to answer concretely without, unless you know, like, Basically what the, what the RL data for a lot of these,…

AI assessment note: “if you gave the models more time, you know, they would actually be able to answer better.”

Q And did you go directly from that to generally this should scale?

A I think I was generally, like, interested in, like, yeah. I think there were a few things, though, like, there was one part of it that we introduced to kind of, we, we observed that there was a bunch of the data that the mall wasn't learning from, and so we proposed another variant of this, where we actually, uh, were like, oh, what if you actually take the ones where it fails, and you, um, basically, like, ask it to reason about, like, why it should have gotten it right, and then you train as if it got it right. Um, and this version Uh, was kind of a way of extending beyond the kind of the parts of the data that it couldn't, that it couldn't see. So if you only train on like the positive examples, then you end up in this kind of like potential minimum where there's just no more data that it can actually solve. And so back then we were like, what if we just, uh, show it the problems that it didn't solve and try to teach it from those. But I guess another thing that other work has done since then is, oh, what if you just sample a lot? Uh, and that, that also Seems to work, uh, in those works.

AI assessment note: “I think I was generally, like, interested in, like, yeah. I think there were a few things”

Q I mean, that's a good transition to, like, what are you doing?

A Yeah. I'd say that it, the main thing is just that, like, As you kind of have these models that, you know, expand in terms of, like, the horizon that they're automating, you know, we have these models, the, the recent, like, or recent-ish IMO results are, like, a kind of a good example of this. You have these models that go on for, like, you know, hours of, you know, reasoning, um, without any kind of human intervention. And this has kind of been an increasing, Uh, measure of success, I would say, for these labs. So, for example, you know, there's this METR, like, meter, like, uh, benchmark that everyone likes to share whenever there's a new model. Uh, and it's like, oh, we went from being able to have these models work for two, like, complete two-hour tasks autonomously without human intervention to two point five-hour tasks, uh, without human intervention. And obviously there's, like, questions of, like, what do those numbers actually mean? Um, and how, like, should we take them, like, kind of at face value? But regardless, this is kind of in, like, the metric that, you know, people are looking at more and more, uh, to measure progress. But, you know, as we kind of, uh, get these models that increasingly, you know, remove people from the interaction, you end up with Basically, people having less say in, kind of, the things that get built. You end up with, like, you know, I th…

AI assessment note: “Basically, people having less say in, kind of, the things that get built.”