Assertion Partly supported AI assessment confidence: 85% certainty 4/5 debate potential 1/5

Zaremba: RL models need three years of real-time play to learn games

Wojciech Zaremba · An AI Primer with Wojciech Zaremba · Y Combinator · May 17, 2017 · at 6:24

OpenAI cofounder Wojciech Zaremba explains the sample efficiency gap between deep reinforcement learning models and human learners playing simple video games.

0:00 / 0:18exact quote · 18.1s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“Well, for instance, in terms of real-time execution, it takes something around three, three, three years of play to learn to play simple games . I mean it, it can be hugely parallelized, therefore it takes a few days to train it on current computers”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

Reinforcement learning algorithms require around three years of real-time gameplay to learn simple Atari games, though training can be parallelized to run over a few days.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

The claim bundles two assertions regarding deep reinforcement learning algorithms (such as DQN and A3C) on the Atari 2600 benchmark as of May 2017: (1) training required roughly three years of real-time gameplay equivalent, and (2) parallelization enabled training to complete in a few days. The second leg is fully corroborated: distributed architectures like Gorila (July 2015) and multi-threaded asynchronous frameworks like A3C (June 2016) demonstrated that parallelizing across clusters or multi-core CPUs reduced wall-clock training times to 1 to 4 days. However, the first leg is an overstatement in terms of continuous real-time execution: the standard Atari 2600 benchmark protocol (200 million raw frames at 60 fps) corresponds to approximately 38.6 continuous 24/7 days (~926 hours) of real-time gameplay, matching ~1.3 to 2.5 years only under human waking play schedules (1–2 hours/day). Three full continuous calendar years would require ~5.68 billion frames.

DeepMind's standard Atari evaluation protocol (Mnih et al., 2013/2015) trained agents on 50 million steps / 200 million raw emulator frames, which equals 38.6 continuous days (925.9 hours) at 60 Hz.
DeepMind's Gorila framework (Nair et al., 2015) and A3C (Mnih et al., 2016) reduced wall-clock training times from 8-14 days on a single GPU down to 1-4 days via parallel actor-learners.

Checked against

[1] toronto.edu [2] arxiv.org [3] academia.edu [4] gershmanlab.com [5] utoronto.ca [6] julien-vitay.net [7] aaai.org [8] bootstrapping.me

"DQN" "200 million frames" "real time" OR "days" Atari "200 million frames" "38 days" OR "real-time" Atari reinforcement learning "Atari" "real time" "years" reinforcement learning "frames" "reinforcement learning" "three years" OR "3 years" "Atari" "Gorila" "A3C" "Atari" frames training time "days" "DeepMind"

assessed as of 2026-09-05

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

Atari games reinforcement learning

⚑ Spot an error? Report it

More from Wojciech Zaremba

Opinion

Zaremba: Universal basic income is the only viable path as automation grows

“So, I believe that we'll have to offer to people a basic income. I super strongly believe that actually that's the only way.”

Wojciech Zaremba May 17, 2017 ▶ 52:06 An AI Primer with Wojciech Zaremba · Y Combinator

Insight

Zaremba: Most business problems can be solved via supervised learning

“Majority of business problems can be framed as supervised learning, and therefore they can be solved with current techniques, as long as we have sufficient number of input examples and what we want to predict”

Wojciech Zaremba May 17, 2017 ▶ 46:59 An AI Primer with Wojciech Zaremba · Y Combinator

Assertion Partly supported

Zaremba: OpenAI has secured $1 billion in total investment

“In total we gather an investment of one billion dollar in the group.”

Wojciech Zaremba May 17, 2017 ▶ 1:34 An AI Primer with Wojciech Zaremba · Y Combinator

Insight

Zaremba: Reinforcement learning struggles in reality due to reward and reset assumptions

“The assumption underlying reinforcement learning Is that then there is some environment, and environment, you are an agent, and you are acting in environment by executing actions and getting rewards from the environment. And the rewards might be taught as, let…”

Wojciech Zaremba May 17, 2017 ▶ 7:12 An AI Primer with Wojciech Zaremba · Y Combinator

Opinion

Zaremba: ImageNet is the essential dataset that enabled deep learning

“That's the essential data set that made deep learning happen.”

Wojciech Zaremba May 17, 2017 ▶ 34:36 An AI Primer with Wojciech Zaremba · Y Combinator

Assertion Supported

Zaremba: ImageNet error dropped to 3%, achieving superhuman vision performance

“Within several years, people got down, I believe, to three percent error, and that's essentially superhuman performance.”

Wojciech Zaremba May 17, 2017 ▶ 37:40 An AI Primer with Wojciech Zaremba · Y Combinator

Zaremba: RL models need three years of real-time play to learn games

More from Wojciech Zaremba

Turn any episode into a week of clips.