Jack Hopkins
Creator, Factorio Learning Environment. On 1 show, 1 appearance. The Shows tab opens the full record on each.
He created the Factorio Learning Environment to enable unbounded evaluation of frontier LLM agents. The framework provides benchmarks for assessing code generation, spatial reasoning, and long-term planning.
1shows
1appearances
8statements
6resolved
5supported
1contradicted
83%fully supported
Everything Jack Hopkins said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Factorio benchmark results show reasoning models underperform expectations on extended planning
“One thing we have found in preliminary results is that the reasoning models don't seem to do as well as you'd expect in this setting. And I think that's probably because the way we set this up, it's a bit like we're already making it do reasoning traces over a…”
Untrained AI models exhibit a 100x competency gap versus human players
“It took models something like eight hours or so to get to the point where they have a kind of working factory that could make a few things, a few let's say iron gear wheels or electric circuits, or maybe some science and maybe start progressing through the tre…”
Claude scored nearly twice as high as the next best model
“So we see that Claude right here is almost got twice the score of the nearest best model.”
Dual reward signals prevent AI agent behavioral collapse in Factorio
“So we have these kind of two reward signals that compliment each other to and the reason why this is necessary is to avoid certain, I guess, behavioral collapses where a model might choose, for example, to mine coal and mine a billion or a trillion coal. And t…”
Only Google models defined reusable code in the Factorio AI benchmark
“Only Google models tend to do this which is quite interesting.”
Claude 3.5 wrote fire-and-forget code while GPT-4 used defensive programming
“Claude, for instance, the Sonnet 3.5 was very much fire and forget. It would write code in a kind of Pythonic way, just like, let it fail. Don't be careful about it. Whereas GPT four would use defensive programming, use self assertions.”
Providing agents with RAG factory blueprints yielded zero benchmark score improvement
“When you try and move that into a benchmark setting with already pre-trained models, just using in-context learning, it's just not that helpful. A thousand lines of Python telling you how to make this kind of factory unit, which it may not be directly applicab…”
Factorio requires one million resources to beat compared to Minecraft's 200
“In order to complete Factorio, you know, complete and launch a rocket, you need to mine maybe about a million raw resources to get to that point. Whereas the same kind of point in Minecraft, you have to just get maybe 200 resources. So, you know, we have order…”
One line per show, most statements first. The link opens Jack's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
 LATENT SPACELEDGER | Creator, Factorio Learning Environment | 1 | 8 | 83% 5/6 | full record on Latent Space → |