The Exchanges

Q For those who haven't gone too deep on it, the way that you present the inside of React, like, can you do that also for reflection?

A Yeah. I think one way to think of reflection is that the traditional idea of reinforcement learning is you have a scalar reward, and then you, you somehow back propagate the signal of the scalar reward. To the rest of your neural network through whatever algorithm, like policy gradient or A to C or whatever. And if you think about the real life, you know, most of the reward signal is not scalar. It's like your boss told you, you know, you should have done a better job in this, but a good job on that or whatever, right? It's not like a scalar reward, like 29 or something. I think in general, human do more deal more with, you know, long scalar reward, or you can say language feedback, right? And the way That they deal with language feedback also have this kind of back propagation kind of process, right? Because you start from this, you did a good job on job B and then you reflect, you know, what could have done different to, to change, to make it better. And you kind of change your prompt, right? Basically you change your prompt on how to do job A and how to do job B. And then you, you do the whole thing again. So it's really like a pipeline of language where it's self-created descent. You have something like tax reasoning to, to replace Those gradient descent algorithms. I think that's one way to think of reflection, yeah.

AI assessment note: “I think one way to think of reflection is that the traditional idea of reinforcement learning”

Q Thoughts is, like, you generate All these possible outcomes and then find the best tree to get to the end. Maybe back to the latency question, you can't really do that if you have to respond in real time. So what are maybe some of the most helpful use cases for things like this? Where have you seen people adopt it where the high latency is actually worth the wait?

A For things that you don't care about latency, obviously, for example, if you're trying to do math, right, if you're just trying to come up with the proof. But I feel like one type of task is more about searching for a solution, right? You can try a hundred times, but if you find one Solution. That's good. Like, for example, if you're finding a mass proof or if you're finding a good code to solve a problem or whatever. And I think another type of task is more like reacting, right? For example, if you're doing customer service, you're like a web agent booking a ticket for like a end user, right? Those are more like kind of reactive kind of tasks, right? You have to, or more real time tasks, right? You have to do things fast. They might be easy, but you have to do it reliably and you care more about like, can you solve Person of the time out of a hundred, but for the type of search type of tasks, then you care more about, you know, can I find one solution out of a hundred? So it's kind of symmetric and different.

AI assessment note: “if you're trying to do math, right, if you're just trying to come up with the proof”

Q really focused on agents. Um, I know you talked before about, you know, it's next token prediction enough to get to like Problem solving. If you say you got the perfect environment, they got the terminal, they got everything. And if you were to now move down to the model level and say, I need to make a model that is better for like agentic workflow, where would you start?

A I think it's data. I think it's data because, uh, like changing architecture now is too hard and we don't have a good, better alternative solution now. I think it's mostly about data and agent data is obviously hard because People just write down the final result on the internet. They don't write down how they, like step by step, how they do the thing on the internet, right? So naturally it's easier for models to learn chain of thought than tool call or whatever agent self-reflection or search, right? Like even if you do a search, you won't write down all the search processes on the internet. You would just write down the final result. And, uh, I think it's a great thing that Lama four is going to be more, Towards agents, that means, I mean, that should mean a lot for a lot of people.

AI assessment note: “I think it's data. I think it's data because, uh, like changing architecture now”

Q If you have a model that can simulate the persona, like the user, the right way, shouldn't the model also be able to accomplish the task, right? If he has the knowledge of, like, what the person will want, then it means...

A This is a great question. I think it really stems from, like, asymmetry of information, right? Because if you think about the customer service agent, it has information that you cannot, you cannot access, right? Like the, the, APIs it could call or, you know, the policies of internal company policy, whatever. And that I think very interesting for TallBench is like, it's kind of okay for the user to be kind of stupid. So you can imagine like there are failure cases, right? But I think in our case, as long as the user specified the need very clearly, then it's up to the agent to figure out, for example, what is the second cheapest flight from this to that under that constraint, very complicated reasoning involved. Like we shouldn't require users to be able to solve those things. They should just be able to clearly express their need. But then if the task failed, then it's up to the agent. That makes the evaluation much easier.

AI assessment note: “I think it really stems from, like, asymmetry of information”

Q Very pro that. Yeah. Before we leave the topic of, uh, trees and searching. Not specific about Qstar, but there's a lot of questions about MCTS and this combination of tree search and language models, and I just had to get in a question there about how seriously should people take this?

A Again, I think it depends on the tasks, right? So MCTS was magical for Go, but it's probably not as magical for robotics, right? So I think right now the problem is not even that we don't have good methodologies, it's more about we don't have good tasks. It's also very interesting, right? Because if you look at my citation, it's like, obviously the most cited are React, Reflection and Triumphs. All those are methodologies, but I think like equally important if If not more important life, my work is like benchmarks and environments, right? Like web shop or sweet venture or whatever. And I think in general, what people do in academia that I think is not good is they choose a very simple task, like Alford, and then they apply overly complex methods and to show the improved two percent, uh, uh, I think like you should probably match, you know, the level of complexity of your task and your method, right? I feel like where tasks Are kind of far behind the, the, the method in some sense, right? Because we have some good test time approaches, like whatever reactor or reflection entry of solid or Like there are many, many more complicated testing methods afterwards, but on the benchmark side, we have made a lot of good progress this year, last year, but I think we still need more progress towards that, like better coding benchmark, better web agent benchmark, better agent benchmark, not…

AI assessment note: “I think it depends on the tasks, right? So MCTS was magical for Go”