Untrained AI models exhibit a 100x competency gap versus human players
Jack Hopkins · ⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins · Apr 27, 2025 · at 24:05
Jack Hopkins discusses benchmark results from testing LLM agents in the Factorio Learning Environment against human performance.
0:00 / 0:20exact quote · 20.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“It took models something like eight hours or so to get to the point where they have a kind of working factory that could make a few things, a few let's say iron gear wheels or electric circuits, or maybe some science and maybe start progressing through the tree. So, you know, there's probably a hundred times difference in the competency of these models untrained than a human expert.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jack Hopkins
Factorio benchmark results show reasoning models underperform expectations on extended planning
“One thing we have found in preliminary results is that the reasoning models don't seem to do as well as you'd expect in this setting. And I think that's probably because the way we set this up, it's a bit like we're already making it do reasoning traces over a…”
Claude scored nearly twice as high as the next best model
“So we see that Claude right here is almost got twice the score of the nearest best model.”
Dual reward signals prevent AI agent behavioral collapse in Factorio
“So we have these kind of two reward signals that compliment each other to and the reason why this is necessary is to avoid certain, I guess, behavioral collapses where a model might choose, for example, to mine coal and mine a billion or a trillion coal. And t…”
Only Google models defined reusable code in the Factorio AI benchmark
“Only Google models tend to do this which is quite interesting.”
Claude 3.5 wrote fire-and-forget code while GPT-4 used defensive programming
“Claude, for instance, the Sonnet 3.5 was very much fire and forget. It would write code in a kind of Pythonic way, just like, let it fail. Don't be careful about it. Whereas GPT four would use defensive programming, use self assertions.”
Providing agents with RAG factory blueprints yielded zero benchmark score improvement
“When you try and move that into a benchmark setting with already pre-trained models, just using in-context learning, it's just not that helpful. A thousand lines of Python telling you how to make this kind of factory unit, which it may not be directly applicab…”