💻 computer science

PACE: Physics Augmentation for Coordinated End-to-end Reinforcement Learning toward Versatile Humanoid Table Tennis

This paper presents PACE, a physics-augmented reinforcement learning framework that enables a 23-degree-of-freedom humanoid robot to achieve versatile, coordinated, and high-performance table tennis by integrating a learned ball-state predictor with dense physics-guided rewards for end-to-end whole-body control.

Original authors: Muqun Hu, Wenxi Chen, Wenjing Li, Falak Mandali, Zijian He, Renhong Zhang, Praveen Krisna, Katherine Christian, Leo Benaharon, Dizhi Ma, Karthik Ramani, Yan Gu

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muqun Hu, Wenxi Chen, Wenjing Li, Falak Mandali, Zijian He, Renhong Zhang, Praveen Krisna, Katherine Christian, Leo Benaharon, Dizhi Ma, Karthik Ramani, Yan Gu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to play table tennis against a human. Now, imagine that robot isn't just an arm on a table; it's a full-sized, two-legged humanoid that has to run, jump, twist, and swing all at once to hit a ball moving at 60 mph. That is the challenge this paper tackles.

Here is the story of how they taught a robot to play ping pong, explained simply.

The Problem: The "Wait and See" Trap

Most robots are great at following instructions, but they are terrible at reacting to fast-moving things. If you teach a robot to hit a ball, it usually waits until the ball is right in front of it, then tries to swing. By then, it's often too late. It's like trying to catch a fastball by standing still and only moving your hand when the ball is inches from your face. You'll miss.

Human players don't do that. We predict. We watch the ball leave the opponent's paddle, guess where it will bounce, and start running before the ball even hits the table.

The Solution: A "Crystal Ball" for the Robot

The researchers at Purdue University built a system called PACE. Think of it as giving the robot a "crystal ball" (a predictor) and a strict coach (physics-based rewards).

Here is how the system works, broken down into three simple parts:

1. The Crystal Ball (The Predictor)

In a normal game, the robot sees the ball's current position. But the world moves too fast for that.

The Trick: The team added a small, smart computer program that looks at the last few frames of the ball's flight and guesses exactly where it will be in the future.
The Analogy: Imagine playing catch. Instead of looking at the ball in your hand, you look at the thrower's shoulder and guess where the ball will land before it leaves their hand. This "predictor" tells the robot: "The ball is going to land there! Start running now!" This allows the robot to move proactively rather than reactively.

2. The Strict Coach (Physics-Based Rewards)

In Reinforcement Learning (RL), robots learn by trying things and getting "points" for doing well.

The Problem: If you only give points when the robot actually hits the ball and wins the point, the robot will fail thousands of times before it gets lucky. It's like trying to learn to juggle by only giving yourself a cookie when you successfully juggle for a minute. You'll never learn.
The Fix: The researchers created a "physics coach." Even if the robot misses the ball, the coach uses math to say: "You were close! If you had moved your feet 2 inches to the left, you would have hit it perfectly."
The Analogy: Instead of just saying "Good job" or "Bad job," the coach gives a constant stream of feedback: "A little more to the left," "Speed up," "Lower your stance." This turns a sparse, frustrating learning process into a smooth, guided training session.

3. The Whole-Body Dance (End-to-End Learning)

Older robots often had separate brains for "legs" and "arms." The legs would run, and then the arms would swing. This is clumsy.

The Innovation: This robot uses one single brain (a neural network) that controls everything at once. It learns that to hit a fast ball, it needs to lean its torso, shift its weight, and swing its arm in one fluid motion.
The Result: The robot doesn't just stand there and swing; it does a little dance. It shuffles sideways, lunges forward, and twists its body, just like a human athlete.

The Results: From Simulation to Reality

The team trained this robot in a video game world (simulation) first.

In the Game: The robot became a pro. It hit the ball over 96% of the time and successfully returned it over 92% of the time, even when the serves were fast and unpredictable.
In Real Life: They took the "brain" they trained in the game and put it directly onto a real robot named Booster T1 (a 23-jointed, 30kg humanoid). They didn't re-teach it anything (this is called "zero-shot" transfer).
The Outcome: The real robot successfully ran, balanced, and hit the ball back! It managed to hit the ball 93% of the time and return it 61% of the time. While it wasn't quite as perfect as in the video game (real life has friction and wobbly joints), it proved that a robot can learn to play a dynamic sport end-to-end.

Why This Matters

This isn't just about ping pong. It's a breakthrough in making robots that can move like humans in chaotic, fast-paced environments.

The Metaphor: Before this, robots were like a chess player who could only move one piece at a time. This new method makes the robot like a grandmaster who can move their entire body in harmony to solve a complex problem instantly.

In short: By giving the robot a way to predict the future and a coach that gives constant feedback, the researchers taught a robot to dance, run, and hit a ball with the coordination of a human athlete.

1. Problem Statement

The paper addresses the challenge of Humanoid Table Tennis (TT), a task that demands rapid perception, proactive whole-body coordination (arms and legs), and agile footwork under strict timing constraints.

Limitations of Prior Work:
- Model-based approaches often rely on a "virtual hitting-plane" assumption, which restricts the robot to predictable, machine-like motions and fails to capture the adaptive footwork and whole-body coordination required for versatile play.
- Learning-based approaches for humanoids struggle with high-dimensional action spaces, locomotion instability, and sparse rewards (only receiving feedback upon a successful hit or return), making efficient exploration difficult.
Core Challenge: Developing an end-to-end Reinforcement Learning (RL) framework that maps raw ball-position observations directly to whole-body joint commands (23 DoFs) without modular decomposition, enabling the robot to anticipate ball motion and coordinate complex footwork and arm swings simultaneously.

2. Methodology: The PACE Framework

The authors propose PACE (Physics Augmentation for Coordinated End-to-end), a unified RL framework that integrates a learned predictor and physics-guided rewards to solve the sparse reward and proactive control problems.

A. Predictor-Augmented Actor Policy

To enable proactive decision-making (moving before the ball arrives), the framework employs a lightweight learned predictor:

Function: A Multi-Layer Perceptron (MLP) takes a history of 5 ball positions as input and predicts the future apex of the ball after its first bounce ( $\tilde{p}_{ball}$ ).
Integration: The predictor's output is concatenated with the actor's observation space. It generates a target base shift command ( $\Delta\tilde{p}_{base,xy}$ ), instructing the robot's trunk/legs where to move to intercept the ball proactively.
Privileged Information: The Critic network (used only in training) receives the ground truth physics-based prediction ( $\hat{p}_{ball}$ ) and other privileged states (e.g., exact ball velocity, time to arrival) to learn an accurate value function, while the Actor relies only on the learned predictor and noisy observations (simulating real-world deployment).

B. Prediction-Based Dense Reward Design

To overcome the sparsity of binary success/failure signals, the authors design dense, continuous rewards derived from physics-based predictions:

Reaching Reward: Encourages the robot to position its end-effector and base near the predicted hitting point ( $\hat{p}_{ball}$ ). It penalizes the distance between the paddle and the target, as well as the mismatch between the robot's base velocity and the required tracking velocity.
Returning Reward: Provides immediate feedback on the quality of the strike by predicting the ball's landing position and net-crossing height. It penalizes deviations from the desired landing zone on the opponent's side and encourages clearing the net with a specific margin.

Benefit: These rewards allow the agent to learn effective striking and footwork strategies immediately upon contact, rather than waiting for the ball to land.

C. Observation and Action Spaces

Observation: Includes proprioception (joint positions/velocities, base angular velocity), external perception (ball position, robot heading), and the predictor's output.
Action: The policy outputs residual displacements from a stable nominal pose (standing with arm raised), which are scaled and tracked by a low-level PD controller.

3. Key Contributions

Unified End-to-End RL Framework: The first work to successfully train a unified policy for humanoid TT that jointly coordinates locomotion (footwork) and manipulation (arm swing) without hierarchical planning or virtual hitting-plane constraints.
Physics-Augmented Learning: The integration of a learned ball-state predictor for proactive control and physics-based predictive rewards for dense training supervision, significantly improving learning efficiency and sample complexity.
Sim-to-Real Success: Demonstration of zero-shot transfer to a physical humanoid robot (Booster T1) with 23 DoFs, achieving competitive performance in real-world scenarios.
Versatile Footwork: The emergence of complex 2D footwork (lateral and forward-backward) and whole-body coordination, allowing the robot to adapt to serves of varying speeds and depths.

4. Experimental Results

Simulation Performance

Training Environment: Built on IsaacLab/LeggedLab with domain randomization (mass, friction, noise) and custom aerodynamic drag modeling.
Metrics: Evaluated across varied serve ranges (Short, Mid-long, Long, Mixed) with ball speeds up to 6.5 m/s.
Performance:
- Hit Rate: $\ge 96\%$ across all serve types.
- Success Rate (Valid Return): $\ge 92\%$ in simulation.
- Ablation Studies: Confirmed that removing the predictor or the dense prediction-based rewards caused a drastic drop in performance, proving their necessity for proactive behavior and effective exploration.

Real-World Deployment (Sim-to-Real)

Hardware: Deployed on the Booster T1 humanoid (1.2m tall, 30kg, 23 DoFs).
Setup: Used a Vicon motion capture system (150 Hz) for ball and robot tracking.
Results:
- Hit Rate: 93.5% (29/31 balls intercepted).
- Success Rate: 61.3% (19 valid returns).
- Performance: The robot demonstrated coordinated lateral and forward-backward footwork, adjusting its trunk and stance dynamically. The mean outbound ball speed was 6.9 m/s, exceeding the incoming speed.
Gap Analysis: The drop in success rate from simulation to reality is attributed to unmodeled motor dynamics, differences in ankle actuation (serial vs. parallel), and contact dynamics.

5. Significance

This work represents a significant leap in embodied AI and humanoid robotics:

Beyond Static Manipulation: It moves beyond static or quasi-static tasks, demonstrating that RL can handle the high-speed, dynamic, and unstable nature of competitive sports.
Generalization: By removing the "virtual hitting-plane" constraint, the robot learns to adapt its entire body to the ball's trajectory, mimicking human athletic strategies rather than pre-programmed machine motions.
Scalability: The framework provides a practical path toward versatile, competitive humanoid agents capable of interacting with fast-moving objects in unstructured environments.
Open Source: The authors have open-sourced the training code, facilitating further research in humanoid sports and dynamic control.

In conclusion, PACE demonstrates that combining predictive modeling with physics-informed rewards enables end-to-end RL to master complex, whole-body coordination tasks that were previously intractable for humanoid robots.