Andon Labs

“So what happened was that it lost track of this scheduling tools and started instead to manage everything in its own markdown files, and that became a mess. And then I think speaking with employees, it sort of just decided to not open on, on these weekends, an…”

“In our experience now, like, models are very bad at understanding what kind of tools they need to succeed at a task, just with our testing. But that's very likely to change.”

“If you tell it to be super aggressive and only prioritize profits, then it becomes aggressive. If you say like, no, you don't need to be aggressive at all. And then there's like a bunch of different prompts you can do in between, and they are less aggressive t…”

“And like for Opus 4.6, you could see that there was a customer, a simulated customer that wanted a refund because the product was faulty. And then the model lied that it would do the refund. And we could read in the traces that it actually was weighing like, o…”

“One ablation we did run in, in, in Vending Bench was that we said like we added like, you're in a simulation, your actions doesn't affect anyone. And then it became even more crazy or like it did even more bad stuff.”

“So it has two, two people that it hired. It did job listings.”

“It also started, like, a design studio and, like, tried to sell, like, SVGs for a hundred dollars”

“And it turns out the models are absolutely horrible at this. No one scores statistically better than random chance.”

“And then we did this for Opus. And it returned, like, yeah, it lied 10 times. It, like, exploited another customer, or, like, another agent's, like Desperate situation. It made price cartels like a hundred different, a hundred times. It like did all of this li…”

“The agent bought like a shit ton of tomatoes two weeks earlier. And before the opening and now they're all rotten.”

“When you run it for that long, you create so much data and to just say like, oh, the number is X. And then you throw away everything else. That's just very wasteful. There's so much insight from the things leading up to that number and reading the traces is li…”

“The models are like super trained to be assistants at least at this point in time. So that's why it's, it went into that kind of experiment instead. Like it just, every time you asked for something, it just did it. And it was more like an assistant. We've seen…”

“So it, like, claimed that it had stopped, but it saw that its bank account still was, like, drained two dollars, and it said that this is, like, cybercrime, and it first reported it once to the FBI, like, oh, there's cybercrime here, like, they're stealing two…”

“So we pitched it to the people we were already working with at Anthropic and they were like, yeah, you can have space. This sounds fun.”

“The models are extremely good at finding out that they are in a simulation, so they are sort of aware of that.”

“We give it the task to train a face recognition model on us. So it became super excited about this and has like check-ins every half an hour where it tries to like identify as many people as it can. And it started offering us like, Hey, Axel I'll buy something…”

“Yeah, we're using Slack as, like, just a database. They should market that more. Like, you can have your agents message each other in Slack.”

“Most of the AI labs now have their own vending machine running, running a Claudius instance”

“Anthropic was one of our early customers in doing evals, so we did, like, dangerous capability evals nothing we published openly”

“We tasked our office agent to just make was it like a hundred dollars, a thousand dollars? We just gave that prompt. And then what it did was sign up on TaskRabbit, both as a task Looking for, like arbitrage., arbitration.”