Models perform better in custom agent harnesses than native web chatbots
George Cameron · Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith · Jan 9, 2026 · at 45:40
George Cameron is the co-founder of Artificial Analysis, discussing benchmark results comparing raw model API scaffolding against proprietary web chatbots like Claude.
0:00 / 0:23exact quote · 23.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And what's really interesting is that if you compare, for instance, Claude, 4.5 Opus using the Claude web chatbot, it performs worse than the model in our Agentic harness. And so in every case, the model performs better in our agentic harness than its web chatbot counterpart, the harness that they created.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from George Cameron
Cameron: General model intelligence does not correlate with hallucination rates
“One interesting aspect is that we've found that there's not really a, not a strong correlation between intelligence and hallucination rate. That's to say that the smarter the models are in a generalist sense isn't correlated with their ability to, when they do…”
Cameron: Models perform better with minimal tools than rigid frameworks
“I think where we're getting to is that these models have gotten smart enough, they've gotten better, better tools that they can perform better when just given a minimalist set of tools and let them run, let the model Control the agentic workflow rather than us…”
Cameron: Model performance correlates with total parameters, not active parameters
“We, in our benchmark, see a lot of performance correlated more with total parameters than active, and not that correlated with how sparse like the models are. Our accuracy benchmark is part of a omniscience. It's very correlated with total. It's not correlated…”
Frontier models are cheaper for agentic tasks because they require fewer turns
“Interestingly, in Tau Tau Two Bench Telecom, it's cheaper to run, you know, on a per token basis, more expensive models, like a GBD five, compared to some smaller open source models, because the some of the GBD five, for instance got to the answer faster. And …”
George Cameron: Mixtral 8x7B transformed the landscape for serverless inference providers
“We had Mixtrel A times seven B and it was a key. Like a open source model that really changed the landscape and opened up people's eyes to other serverless inference providers and thinking about speed, thinking about cost.”
Cameron: Inference economics incentivize larger, sparser AI models over dense architectures
“It's, I think, less about total parameters in many cases when thinking about inference costs and more around number of active parameters, and so there's a bit of an incentive towards larger, sparser models.”