🤖 AI

Physical Atari: A Robust and Accessible Platform for Real-time Reinforcement Learning on Robots

This paper introduces "Physical Atari," a robust and affordable $1,000 platform that uses a custom robot to physically control Atari games, enabling real-world reinforcement learning experiments that demonstrate the critical need for on-device adaptation to handle distribution shifts between training and deployment.

Original authors: Khurram Javed, Joseph Modayil, Gloria Kennickell, Richard S. Sutton, John Carmack

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Khurram Javed, Joseph Modayil, Gloria Kennickell, Richard S. Sutton, John Carmack

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to play video games. Usually, scientists do this in a computer simulation, like a video game inside a video game. But the real world is messy, unpredictable, and doesn't wait for you to think.

This paper introduces a new, affordable, and tough system called Physical Atari. Think of it as a "robotic arm" that physically presses the buttons on an old-school Atari joystick to play games on a real screen, all while a computer watches and learns from the results.

Here is the breakdown of how they built it and what they found, using some everyday comparisons:

1. The Two Main Characters

The system is made of two custom-built gadgets:

The Robotroller (The Muscle): This is a small robot built to hold and move an unmodified Atari joystick.
- The Problem: If you build a robot out of cheap plastic parts and make it move fast, it usually breaks or wears out quickly, like a cheap toy that falls apart after a week of rough play.
- The Fix: The team used ball bearings (the same kind found in skateboard wheels) for every moving part. This makes the robot glide smoothly instead of grinding against itself. They also swapped out plastic gears for metal gears and added a "reflex" system.
- The Reflex: Imagine if your hand jerked back automatically when you touched a hot stove to prevent a burn. The robot has a similar safety switch. If a motor tries to push too hard or gets stuck, the system instantly cuts the power for a split second to save the motor from burning out. This allowed the robot to run for weeks without breaking.
The Atari Devbox (The Brain & Screen): This is a small computer box with a screen.
- It runs the actual Atari games (like Pong or Ms. Pac-Man).
- It doesn't just show the game; it also flashes special invisible codes (called AprilTags) on the screen. These codes tell the robot: "You just got a point!" or "Here is exactly where the game is on the screen."

2. How They Talk to Each Other

The setup works like a relay race:

The Eye: A standard webcam watches the screen.
The Brain: A computer looks at the video, figures out what the game is doing, and decides what move to make next.
The Hand: The computer sends a signal to the Robotroller.
The Action: The Robotroller physically moves the joystick to press the buttons.
The Loop: The camera sees the new game state, and the cycle repeats.

The whole process takes about 165 milliseconds (a little over a tenth of a second). That's about as fast as a human reacts to something, meaning the robot isn't "cheating" by thinking faster than reality allows.

3. The Big Discovery: "The Body Matters"

The researchers wanted to see if an AI could learn directly on a robot without needing a simulator. They ran the robot for weeks, playing games non-stop, and it worked perfectly.

But then, they did a tricky test:

They taught a robot (let's call him Robot A) how to play Pong for six hours.
They took the "brain" (the learned strategy) from Robot A and put it into Robot B. Robot B was built exactly the same way, but it was a different physical unit with slightly different parts.

The Result: Robot B played much worse than Robot A.
Even though the robots were built to be identical, tiny, invisible differences in their physical bodies (like a slightly tighter screw or a tiny bit of friction in a bearing) messed up the timing. In a game like Pong, where you have to hit a ball at the exact right millisecond, being off by a tiny fraction of a second is the difference between winning and losing.

4. The Lesson

The paper concludes that learning directly on the specific robot you will use is crucial.

If you train a robot in a perfect computer simulation or on a different robot, even tiny physical differences can ruin its performance when you finally put it in the real world. The best way to get a robot to be good at a task is to let it learn and adapt while it is actually holding the joystick and playing the game itself.

In short: They built a cheap, durable robot that can play Atari games for weeks without breaking. They proved that to get the best results, you can't just copy-paste a robot's "brain" to a new body; the robot needs to learn on its own specific body to handle the tiny, real-world imperfections.

Technical Summary: Physical Atari

Problem Statement

Reinforcement learning (RL) research has largely relied on turn-based simulations where the environment waits for an agent's response before evolving. This contrasts with the physical world, where the environment evolves continuously regardless of the agent's decision-making latency. This setting, termed real-time reinforcement learning, presents unique challenges, including the need for rapid decision-making to minimize the gap between perceived and actual states, and the ability to vary computational effort based on the cost of delay.

While RL in robotics is a growing field, most approaches rely on sim-to-real transfer (learning in simulation and deploying on robots) or learning from human demonstrations. The third approach—learning directly on robots—offers significant advantages, such as eliminating the need for simulators or human data and avoiding distribution shifts between learning and deployment environments. However, the paper identifies a critical gap: there is currently no platform for studying real-time RL directly on robots that is both reliable (capable of long-duration operation without failure) and accessible (affordable and easy to construct).

Methodology: The Physical Atari Platform

To address this gap, the authors built Physical Atari, a system comprising two primary components: the Robotroller and the Atari Devbox.

1. The Robotroller

The Robotroller is a robot designed to actuate an unmodified, commercially available Atari CX40+ joystick.

Mechanical Design: To ensure durability for weeks of non-stop operation, all moving parts utilize 608ZZ ball bearings to reduce friction and wear on 3D-printed PLA components. The design features two servo arms whose axes of rotation align with the joystick's pivot point, allowing for coordinated movement without binding.
Actuation: It employs three Dynamixel XC-330 servo motors (chosen for their metal gears over the plastic gears of the cheaper XL-330) to control vertical, horizontal, and fire actions. The servos are connected via a Waveshare Serial Bus Servo Driver Board.
Robustness Enhancements:
- Thread Locking: Loctite 243 was used to prevent screws from loosening during extended use.
- Motion Smoothing: The authors tuned the servos' PID controller parameters (reducing P gain, increasing I gain) to eliminate jerky movements that stressed the controller.
- High-Current Reflex: To prevent servo damage from current spikes when a setpoint is missed, a high-frequency loop monitors current draw. If a threshold is exceeded, the servo's torque is disabled for one millisecond, acting as a protective reflex similar to biological systems.
Cost & Accessibility: The Robotroller can be built for approximately $400 in parts, with the remaining components (3D printed chassis) manufacturable on consumer-grade printers like the Bambu Lab P1S.

2. The Atari Devbox

The Atari Devbox serves as the environment interface.

Hardware: It uses a Raspberry Pi 5 and a 5-inch Waveshare display. A custom chassis houses the components.
Interface: A male DB9 connector is wired directly to the Raspberry Pi's GPIO pins to read signals from the CX40+ joystick.
Functionality: A C++ program reads GPIO states with sub-millisecond latency, maps them to the 18 discrete actions of the Arcade Learning Environment (ALE), and renders game frames at 60 FPS.
Visual Communication: The screen displays AprilTags: four corner tags for screen localization by the agent's camera, and dynamic tags on the side to encode the reward signal.

3. System Integration and Latency

The full system includes an off-the-shelf Razer Kiyo Pro camera (streaming uncompressed video via USB 3.1) and a desktop computer (Framework with AMD Ryzen AI Max+ 395).

Latency: The end-to-end response time (rendering, capture, processing, actuation) was measured at approximately 165 ms, comparable to typical human reaction times. This latency does not include the time required for a deep neural network to select an action.

4. Learning Algorithm

The experiments utilized a Reactive Reinforcement Learning formulation.

Architecture: A fully convolutional network processes visual input (128×128 frames blended into an exponential moving average).
Action History: A distinguishing feature is the late fusion of action history. The agent encodes recent actions as a factored binary vector (vertical, horizontal, fire) and concatenates this with the visual feature maps after the first convolutional layer. This allows the value estimator to be explicitly conditioned on both smoothed visual observations and recent control decisions.
Training: The agent uses n-step returns, an $\epsilon$ -greedy policy, and a replay buffer that prioritizes recent online transitions. Optimization uses AdamW for convolutional layers and SGD with momentum for the final linear layer.

Key Results

1. Robustness and Reliability

The system was tested over 145 hours of continuous learning across six Atari games (Pong, Seaquest, Ms. Pacman, Assault, Asterix, Kangaroo).

Outcome: The agent successfully learned on all games without any mechanical failures or human intervention. The performance was consistent across multiple runs, validating the durability of the bearing-based design and the protective reflex mechanisms.

2. Sensitivity to Distribution Shifts

The authors investigated how policies learned on one Robotroller transfer to a different, identically built Robotroller.

Outcome: Policies consistently performed worse when deployed on a different robot body. The degradation was most severe in games requiring precise timing (e.g., Pong), where even minor physical discrepancies caused the agent to miss the ball by small margins.
Continual Adaptation: When the learning algorithm was allowed to continue adapting on the new body after a switch, performance recovered and improved, demonstrating the necessity of on-device adaptation.

Significance and Claims

The paper claims that Physical Atari is the first robotic platform that is both affordable (under $1,000 total) and durable enough for weeks of non-stop real-time reinforcement learning.

The authors emphasize that this platform enables the study of challenges inherent to the physical world that are often ignored in simulation, such as:

Time delays and non-stationarities.
The impact of distribution shifts caused by inevitable physical differences between robot bodies.

The results underscore that even minimal discrepancies between learning and deployment environments can significantly degrade policy performance. Consequently, the paper argues for the importance of on-device adaptation for achieving strong performance in robotics, suggesting that learning directly on the robot (rather than solely in simulation) may be critical for handling real-world variability. The platform also offers a path to systematically compare the three paradigms of RL in robotics: simulation-based learning, learning from human data, and learning directly on the robot.