Dual reward signals prevent AI agent behavioral collapse in Factorio
Jack Hopkins · ⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins · Apr 27, 2025 · at 7:15
Jack Hopkins discusses designing evaluation metrics for AI models playing Factorio within the Factorio Learning Environment.
0:00 / 0:26exact quote · 26.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So we have these kind of two reward signals that compliment each other to and the reason why this is necessary is to avoid certain, I guess, behavioral collapses where a model might choose, for example, to mine coal and mine a billion or a trillion coal. And that's all it does. And that would give a good score. Like that, that would have a high production score. But it would probably indicate something has gone very wrong in the evaluation or training process.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jack Hopkins
Factorio benchmark results show reasoning models underperform expectations on extended planning
“One thing we have found in preliminary results is that the reasoning models don't seem to do as well as you'd expect in this setting. And I think that's probably because the way we set this up, it's a bit like we're already making it do reasoning traces over a…”
Untrained AI models exhibit a 100x competency gap versus human players
“It took models something like eight hours or so to get to the point where they have a kind of working factory that could make a few things, a few let's say iron gear wheels or electric circuits, or maybe some science and maybe start progressing through the tre…”
Claude scored nearly twice as high as the next best model
“So we see that Claude right here is almost got twice the score of the nearest best model.”
Only Google models defined reusable code in the Factorio AI benchmark
“Only Google models tend to do this which is quite interesting.”
Claude 3.5 wrote fire-and-forget code while GPT-4 used defensive programming
“Claude, for instance, the Sonnet 3.5 was very much fire and forget. It would write code in a kind of Pythonic way, just like, let it fail. Don't be careful about it. Whereas GPT four would use defensive programming, use self assertions.”
Providing agents with RAG factory blueprints yielded zero benchmark score improvement
“When you try and move that into a benchmark setting with already pre-trained models, just using in-context learning, it's just not that helpful. A thousand lines of Python telling you how to make this kind of factory unit, which it may not be directly applicab…”