The Exchanges

Q Great. We'll go into, um, all of this in a, in a minute, but before doing so, uh, we, we alluded to, to some of your background. Let's go into it. Starting from the, from the beginning, what was your path to becoming a top researcher?

A I grew up in Russia, in Moscow. Starting from like middle school, high school, I was really interested in mathematics, and I was thinking I will be a mathematician or, or engineer of some kind. I was interested in machines, uh, and, and eventually, uh, computers, and I got into an undergrad in computer science, and I was still thinking that I'll be doing some kind of theoretical, you know, applied linear algebra, tensor methods, things like that. Uh, but at some point, I, Kind of discovered, uh, machine learning. There was this, uh, professor that we had, uh, Dmitry Vetrov, who had one of the, like, leading labs in machine learning in Russia at the time, and I was lucky enough to join that lab and start doing some research on machine learning in my undergrad. So that was around 2013, maybe 2014. I initially was working on non-neural network machine learning, uh, methods, so Gaussian processes. That's kind of By now, you know, nobody really talks about that anymore, but eventually I, I got into a PhD thinking I would still be doing a Gaussian process, but I ended up working on deep learning, and that was actually quite, I'm happy that I didn't work on Gaussian process. I worked on some things related to kind of core machine learning, methodology, optimization, probabilistic methods, questions related to generalization and how the models learn features. After I finished my PhD, I…

AI assessment note: “I was lucky enough to join that lab and start doing some research on machine learning”

Q And then, uh, within that world, your work specifically has focused on weak to strong. Uh, can you explain what that is?

A That is, uh, the project that we did, uh, back at OpenAI. That Work was focusing on the future scenario when we will be trying to align models that are above our own capability on certain tasks. Already now, if you take the frontier LLMs, they are extremely capable, and on a lot of domains, we need expert humans to be able to tell which responses are good, which are correct, which are not correct. But in the future, we are imagining we will have models that are More capable than humans, and even expert humans will not be able to reliably grade very complicated answers from the model. So imagine you ask it to make a repo for you for some, like, you know, new startup idea and just implement it from scratch entirely, and then it gives you, you know, 10,000 lines of code. You have no way of checking if all of this code is correct, if all of this code is safe to use. And so that's the problem of supervision. We are moving to this future when Like it's very hard for a human to supervise, uh, the models, uh, directly. And so instead we studied a simplified setting where we used a small model to try to supervise a larger model.

AI assessment note: “we used a small model to try to supervise a larger model”

Q to 15 years is, uh, like all the brains from academia have been sucked into industry, um, and, uh, you sort of doing the opposite, or maybe both at the same time. Curious for the context, is that more of a personal thing, uh, because you always wanted to do academia, or is there something deeper about, like, the kind of work that you can do in academia versus industry?

A Yeah, it's more about the kind of work. Industry is really great at executing on Ideas, uh, and it's maybe not as good at, like, exploring diverse ideas. Even at the scale of Anthropic OpenAI, uh, there is a lot of focus, uh, in the companies, and there isn't a lot of bandwidth to do exploration, and that has been working extremely well so far. We still probably have a lot of low hanging fruit, uh, left, uh, to, like, get the models to be much better, but I personally find it really Exciting to do more exploratory work and to try things that are different. And for that, I feel like having my own lab in the university is just a better tool.

AI assessment note: “Yeah, it's more about the kind of work.”

Q Why do models do that, or why are they able to do this? Is that basically part of the pre-training, and they effectively learned Being deceptive from us by, by being taught all the deceptive ways humans have behaved over the centuries?

A It's a very interesting question. And, um, yeah, it is quite surprising, actually, that the models would, uh, behave that way after going through some of the alignment training. We don't really know what's the source of this type of behaviors, but that's also true for a lot of other behaviors in the models with, like, even the good ones. We don't really, we cannot always pin down, like, where they come from in the pre-training. I think at least part of it is probably The models, uh, seeing descriptions of AI, like in the science fiction literature going rogue, uh, and, uh, like, yeah, that probably affects how the models behave in similar scenarios. Uh, so for example, in that anthropic study, they have this blackmail scenario where it's kind of really well structured so that the model sees some information about like a CEO of a company that, uh, the CEO is involved in some extramarital, uh, Affair. And then like soon after the model observes that it will be shut down. And then the model kind of puts the two things together and it says, okay, I need to use the first information to prevent me from being shut down. So there is, uh, in the blog post, they note that there is this possibility of like a check of scan that like in the text on the internet, probably if two things co-occur close to each other, then it is likely that they are related to each other. And the model is a sta…

AI assessment note: “I think at least part of it is probably The models, uh, seeing descriptions of AI”

Q Very interesting. All right. Uh, let's switch to, uh, reasoning. Clearly, uh, was a huge year, uh, from that perspective, just massive progress in reasoning. Where do you think we are in that arc? Uh, and what are you excited about on the reasoning front for?

A The biggest step change in the models over the last few years was the reasoning NRL. We have made a lot of progress, and the progress was very fast, uh, in the beginning, where, you know, there was O-one, but then very quickly after that, there was O-three, and on a lot of benchmarks, the, the progress has been extremely dramatic. I remember when we, like, early in the project of the O-one, uh, there was some discussion of, like, will it solve IMO problems, and that seemed kind of Very unlikely to me, but then, yeah, here we are, it can easily solve a lot of IMO problems. So I think we, like, as a community, there was a lot of progress. I think it's, as with many methods, uh, it's starting to be harder to make progress, or at least visible progress. So kind of similar to pre-training, uh, there is still a lot of progress, but it's, the models are already so good that it's kind of harder to see What changes from one to the other as a user of the model? And I think that's also to some extent true for, uh, for the reasoning now, but they're still increasing the scale of the RL, more environments, more, um, more compute spent and, um, yeah, models are still getting more consistent and better. And I think we are at the stage where if we define a benchmark and we can make a relevant RL environment, then we can kind of max it out Uh, pretty quickly, and so we are going through benchma…

AI assessment note: “I think we are at the stage where if we define a benchmark”

Q Let's talk about some of your work in alignment. You mentioned scalable oversight a bit earlier. What does scalable oversight mean?

A Scalable oversight is generally the field of alignment, which deals with using models to assist us in Aligning or other models. So in particular with grading the outputs of other models and checking them for safety or for correctness. It's, it's an interesting field. It's very relevant to both alignment and to capabilities. So generally in machine learning, in RL in particular, we have this problem of supervision. The whole RL relies on our being able to tell which Samples from the model are good versus which are bad. Math with a numerical answer, you can just check the answer, or in competitive coding, you can just check whether the code is passing the tests, and that's why we have seen a lot of progress in those domains. But in creative writing, for example, it's very hard to programmatically tell if one sample is better than the other, and historically people have used this RLHF framework, reinforcement learning from human feedback, but also we now we want to use Models to be able to grade responses of other models to provide critiques or feedback. And then there is a question of how do you use that feedback? How do you learn from the feedback? But yeah, the scalable oversight kind of deals with all of those questions. So using models to, to critique, to provide feedback, to supervise other models.

AI assessment note: “using models to assist us in Aligning or other models”

Q Do you want to maybe define mechanistic interpretability?

A Yeah. Mechanistic interpretability generally, uh, tries to at a low level understand what is happening Inside the model. So they are trying to find these things called circuits that are, you know, some parts of the model that you can isolate and understand and kind of model in your brain that correspond to certain behaviors in the models. And there over the last maybe three years, I think there has been some pretty major progress there. So we are still pretty far from the dream that we will Fully understand everything that happens in the model, but these tools are becoming increasingly more useful internally, uh, at Anthropic in particular, and also there is constant progress, and, uh, it's pretty fascinating work, actually.

AI assessment note: “tries to at a low level understand what is happening Inside the model”

Q Why is it so hard to truly understand what a deep learning model of, of any kind actually does?

A Deep learning models are huge. They have billions, trillions of parameters, uh, and They are doing some messy mathematical computation. You can understand what they're doing at some level. It's like a bunch of matrix multiplications and some, you know, rearrangement of vectors, but that's not a sufficient level of understanding. We want to understand it at a lower level, and it is very possible that that's just not fully possible. Like, it is some computational process that leads to some results. It doesn't have to be the case that you can Kind of describe it in human terms, uh, and kind of understand it very discreetly. I think also, uh, something that contributes to this complexity is just how many things the models are capable of doing. And they are not trained on some small isolated behavior in some small context. They are, you know, they know all of the internet. So all of the information in all languages is somehow encoded somewhere in the weeds. And then they also have all of these behaviors, all of these correlations between the knowledge. All of that is somewhere in the model and just like making sense of all of that is extremely hard.

AI assessment note: “Deep learning models are huge. They have billions, trillions of parameters”

Q question in, uh, all the current reasoning progress. Do we understand what does what and was it, what is responsible for what kind of progress? So if you take, uh, test time compute If you take, uh, the ability to, to search, uh, if you take RL, uh, do, do we know, uh, which one of those techniques, uh, we should turn the knob on, uh, to get better results?

A All of those techniques, they don't exist independently, right? Uh, RL is mainly kind of used to teach the model to use test time compute. Uh, so you first need to prime the model to, to set it up so that it outputs a bunch of tokens before outputting to answer. But then you, you spend the compute in RL so that it learns to output the right tokens. So in my mind, those two are almost kind of indistinguishable, the RL and the test time compute. RL is a method for training and test time compute is maybe just a more general concept. Yeah, you can potentially get to models that use test time compute without RL, but that's not how we are training them right now. Yeah. So I think the trend has been in spending more and more compute on the RL. And getting the models to make better and better use of test time compute. And the tools are also, of course, extremely important, like the web search that you mentioned, and also just the models being able to write Python code and run them, produce artifacts for you. That is extremely important for the product and for making the models useful to people. Conceptually, I think that's a little bit secondary. Like in my mind, the main thing is, you know, the, the RL and getting the models to, to think for a long time.

AI assessment note: “All of those techniques, they don't exist independently, right?”

Q What's, uh, working so far. So, you know, you, you hear talking to people, um, you know, some people talking about like running agents for like a couple of hours, uh, but then some people are talking about like agents running for like 24 hours, uh, 32 hours. Where are we in that arc? And, um, What is working? What is not yet working?

A There is this famous, uh, meter, uh, plot, which shows how long of a task AI is capable of, um, robustly automating, and it's been kind of consistently doubling at that time, uh, every half a year, I think, uh, and it's now in like some hours, uh, so maybe a couple hours. In terms of the methods that are working well, I think, yeah, right now it would involve some kind of a harness, uh, with a bunch of Agents that interact, uh, or, uh, that sequentially solve the task, and there has to be some kind of orchestration, uh, or maybe like some initial task decomposition, and yeah, and it's all not very well established, I'd say. It's, it's a new domain, and, uh, I think we are still figuring out how to best do it.

AI assessment note: “it's now in like some hours, uh, so maybe a couple hours.”

Q I know that you spend some time thinking about, um, the impact of AI on science and math. Um, same idea. Any predictions, uh, there? Like, do you expect, uh, important new discoveries to be made by AI solely by AI?

A Uh, it's a great question. And, uh, I think it's, In sciences, I think that's maybe a little bit more likely. Uh, although I, I also don't know very much about, you know, the life sciences. It feels that there, some discoveries can be made by potentially combining results from different parts of the literature and like proposing some ideas that turn out to, to be true. I think it's hard to imagine the AI making a discovery independently in like Add a main where you need experiments, because my understanding of a lot of science is it's about doing the experiments and you, you need some reasoning to guide what experiments you do, but you also need a lot of iteration and a lot of, uh, like actual, you know, things happening in the physical world. Uh, and at least for now, the AIs are not capable of doing that. In the mathematics, um, I think we will see the models getting better on proving technical Um, results, technical lemmas, uh, maybe including formalization and, uh, like things like lean, uh, the formal theory, improving language. I think the models, it's easy to imagine the models becoming better than humans at proving this technical lemmas, uh, quickly. I think the impact on mathematics is very interesting. Um, so it is improving the output of humans. Already, but it also introduces some noise, right? It also, like some of those rules will be incorrect, and they will be in…

AI assessment note: “hard to imagine the AI making a discovery independently”

Q Okay. Thanks for that. So let's, let's go back, um, to alignment and go a little deeper. Is reasoning a good thing or a bad thing for alignment? You could, uh, argue that, uh, on the one hand, uh, it has more time to not do the wrong thing, but equally it has more time to do the wrong thing. So like, which one is it?

A Yeah, that's a great question. Um, I think the high level answer is that in my mind, at least the risks are associated with the models being more capable. So anything that makes the models more capable is also like making alignment more important and harder. Definitely, you know, the reasoning is an RL is the, the thing that made the models more capable in the last few years. It also provides some new opportunities for alignment. So for example, um, the chain of thought Faithfulness and chain of thought analysis have been like one of the things that people are pushing on. You can read the some trace of what the model is doing. I'm personally a little bit unsure if that's a good, uh, you know, long-term direction for alignment. It seems like as soon as we start kind of applying some optimization pressure, the models will learn to hide what they're doing from the chain of thought. And also we like already have evidence that the models can say one thing in the chain of thought and, but then Actually do something completely different.

AI assessment note: “anything that makes the models more capable is also like making alignment more important and harder”

Q The amount of data or the efficiency of the data?

A It doesn't lead to having more information in the data like that the model can extract. But we argue that that's just not correct. And because the models are, like, it would be true if the model is, has infinite compute. So if the model can fully understand what the deterministic transformation is, then it's not going to be able to extract more information from the transformed data than it used to extract from the original data. But with a limit on the compute, it's actually very possible to apply deterministic transformations to the data. And create information through that. So we have the example of AlphaGo, actually, or AlphaZero in the paper. AlphaZero doesn't use any human data from the perspective of Kolmogorov complexity or the Shannon information theory. It doesn't create information. So it's unclear what is actually learned by the model because it's trained on no data. It can only learn kind of the rules of the game, and that's the only thing. But from this perspective, because the model is computationally bounded, it cannot do the full unroll, full rollout of all the possible games of Go or chess and figure out what's the best move in every possible position. It is actually, there is structure that is produced through this deterministic process, and it is, the model is able to learn that structure. And so we are trying to kind of reconcile these different observations…

AI assessment note: “come up with the notion of structural information Uh, that is dependent on the amount of compute”

Q And that's a new word, right? That's a new term entirely. Among other, uh, things invented a new word in the dictionary. Congratulations on that. So, uh, walk us through the, the whole idea at a high level.

A I guess I want to quickly give a shout out to my collaborators on this work. The lead authors are Mark Finzi. Mark is actually currently at OpenAI, uh, working on synthetic data there. Uh, but we, we were doing our PhD together and our PhD advisor is also on the paper, Andrew Wilson. Uh, but then also, um, there is Shikai and, um, Eding, who, uh, are other, yes, students on the, on the paper. And then Zika Colter, uh, is who's a professor at CMU. He's on the board at OpenAI. He's also, yeah, on the paper. Core idea is to think about how the data can look different for an observer, depending on how much compute the observer has. You can imagine that there is some complicated process that generates the data and A very, very smart observer that has a lot of compute can fully understand what that data is, understand every aspect of it, but a weaker observer that cannot fully model the data, some parts of the data will look like noise to it, and so the amount of structure that you'll see in the data will depend on how much compute you as an observer have, and actually in some cases you can see more structure if you have less compute, which is kind of interesting.

AI assessment note: “Core idea is to think about how the data can look different for an observer”

Q One of the key questions is transformers as a paradigm get us there, uh, or do we need something completely different, like, uh, world models?

A I think the current approach that the companies are taking is, uh, kind of, Yeah, brute force. So kind of, we, we try to come up with as many environments as we can, and like all of the types of tasks that humans are doing, and turn all of them into environments, and then do RL on all of them, and hopefully it generalizes. Of course, pre-training is an example where there has been pretty amazing generalization, where like we train on all of the internet, but we see the models doing like very useful, very practical things, and Some things that are clearly outside of what was in pre-training. The goalpost for what translation should be doing is always moving, but it still seems unsatisfying to me, and I think it's possible that we need new ideas and new methods of training. In the companies, like, people often think about ideas and methods as compute multipliers. Doing this new method is equivalent to spending more compute with the old method, so it kind of saves your compute, and that's kind of how we often think about Ideas, methods, or data. I think there are still major, like, compute multipliers, major ways of saving compute that can lead to better performance without just naively scaling.

AI assessment note: “I think it's possible that we need new ideas and new methods of training.”

Q If the models are getting smarter, particularly if there is a self-awareness, as we discussed, how do evals need to evolve?

A It's an interesting point, like in that blog post, uh, that we discussed at the beginning, it sounds like this is already a major issue, uh, the sandbagging. I think in my understanding, that's mostly A concern that we have, but not necessarily a huge practical issue at the moment. That is something that Anthropic always includes in the system cards for the models. When it releases, there is a section on sandbagging, and we do have some scenarios when the model can artificially suppress its capabilities and perform worse on an evil than it should be. But that's not like a prevailing behavior. I'm working mostly on capabilities and On the capabilities of evals, like the Sweebench, the AME, we are not seeing this, or if we are seeing this, we, we just treat it as under-elicitation. We are not eliciting the best capability possible out of a model, and that can happen for many reasons, like maybe the prompt is suboptimal, or something about the kind of hardness of the eval is suboptimal. But yeah, I think you can, going back to what's in that blog post, you can Kind of manufacture situations or even like discover them naturally where the model identifies that it is being evaluated and then changes behavior based on that. And for example, if it's clear that it's in some kind of fake artificial scenario, it can guess that it's being evaluated, uh, or if you explicitly tell it that it…

AI assessment note: “It's not at the moment a very major concern in my mind.”