💻 computer science

Learning Visuomotor Policy for Multi-Robot Laser Tag Game

This paper proposes an end-to-end visuomotor policy for multi-robot laser tag that leverages knowledge distillation from a multi-agent reinforcement learning teacher and specialized architectural designs to outperform classic modular methods in accuracy and collision avoidance while enabling real-world deployment.

Original authors: Kai Li, Shiyu Zhao

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kai Li, Shiyu Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes game of Laser Tag, but instead of humans running around with plastic guns, it's a team of tiny robots trying to tag each other with laser beams. Now, imagine these robots can't talk to each other, they don't have a GPS map of the room, and they can't use expensive 3D scanners to see how far away things are. They only have a single camera on their "head" (like a human eye) and a brain that needs to make split-second decisions.

This is the challenge the researchers in this paper tackled. Here is the story of how they taught these robots to win, explained simply.

The Problem: The "Modular" Robot vs. The Human Player

Traditionally, engineers build robots like a Swiss Army Knife: they have separate tools for separate jobs.

The Eyes: Look at the image and find the enemy.
The Brain (Math): Calculate exactly where the enemy is, how fast they are moving, and where they will be in 0.5 seconds.
The Map: Build a 3D model of the room to avoid walls.
The Hands: Move the robot and aim the gun.

The Problem: This approach is fragile. If the math for "where the enemy is" gets slightly wrong (because the camera is blurry or the enemy moves weirdly), the whole robot freezes or crashes. It's like trying to drive a car while constantly stopping to calculate the exact distance to every other car using a ruler. It's too slow and prone to errors.

The Human Way: Think about how you play a First-Person Shooter video game (like Call of Duty or Fortnite). You don't stop to calculate the enemy's velocity vector. You just see the enemy, feel the rhythm of the game, and react. You don't need a map; you just know "that red blob is bad, I need to shoot it."

The Solution: The "Teacher" and the "Student"

The researchers decided to teach their robots to play like humans, not like calculators. They used a clever two-step training method called Privileged Imitation Learning.

Step 1: The "God-Mode" Teacher

First, they created a super-smart robot (the Teacher) that had "God-mode" powers.

Superpowers: The Teacher could see the exact coordinates of every robot, every wall, and every obstacle. It knew the enemy's speed and position perfectly.
Training: They let this Teacher play thousands of games against itself using a technique called Reinforcement Learning (trial and error). The Teacher learned the perfect strategy: "If the enemy is here, move there and shoot now."
The Result: The Teacher became a grandmaster of Laser Tag, but it relied on information real robots don't have.

Step 2: The "Blind" Student

Next, they created the Student robot. This is the one that will actually go into the real world.

The Limitation: The Student is "blind" to the math. It only has a camera. It sees a blurry, 2D picture of the room. It doesn't know the enemy's speed or exact distance.
The Lesson: The Student watches the Teacher play. It looks at the camera image and tries to copy the Teacher's moves.
The Magic Trick: To help the Student understand the 2D picture better, the researchers gave it two special "lenses":
1. The Heatmap: Instead of just showing a box around the enemy, they painted a glowing "heat spot" on the screen where the enemy is. This tells the robot, "Aim right here, in the center of the glow."
2. The Depth Map: They used AI to guess how far away things are just by looking at the 2D image, turning it into a black-and-white map where white is close and black is far.

The Student learned to combine these visual clues to mimic the Teacher's perfect moves, even without knowing the exact math.

The Secret Sauce: "Permutation Invariance"

One cool technical detail is how the robot handles multiple enemies or obstacles.
Imagine you are in a room with three friends. If you swap their names, you still see three friends. But a standard computer might get confused if the list of friends changes order.
The researchers built a Permutation-Invariant Feature Extractor. Think of this as a soup blender. It doesn't matter if you put the carrots in before the potatoes or vice versa; the soup tastes the same. Similarly, the robot looks at all the obstacles and enemies as a "set" of objects. It doesn't care which one it saw first; it just understands the scene as a whole. This makes the robot much more stable and less likely to panic when things move around.

The Results: Winning the Game

When they tested this system:

Accuracy: The robot hit its target 16.7% more often than the old "Swiss Army Knife" robots.
Safety: It crashed into walls 6% less often.
Real-World Test: They put this brain onto a real robot with a small computer (like a high-end gaming laptop for robots) and it worked perfectly in a real room with real obstacles.

The Big Picture

This paper is a victory for simplicity and intuition.
Instead of building a robot that tries to be a super-computer calculating physics equations, they built a robot that learns to see and react like a human gamer. By using a "Teacher" with superpowers to train a "Student" with just a camera, they created a system that is cheaper, faster, and much better at the game than the old ways.

It's the difference between a robot that stops to do calculus before shooting, and a robot that just sees the target and shoots—just like you do in a video game.

1. Problem Statement

The paper addresses the challenge of multi-robot laser tag, a simplified yet practical shooting-game scenario where teams of mobile robots must locate, aim at, and shoot opponents while avoiding obstacles and teammates.

Limitations of Classic Modular Approaches:
Traditional methods for such tasks typically follow a modular pipeline (detection $\to$ state estimation $\to$ planning $\to$ control). The authors identify three critical failure points in these approaches:

Observability Issues: Accurate enemy state estimation (position/velocity) from monocular vision requires specific motion patterns to satisfy observability conditions. In dynamic adversarial settings, these conditions are often unmet, leading to unreliable estimates and poor shooting accuracy.
Hardware & Localization Dependencies: Many methods rely on global localization (e.g., motion capture systems) or depth-sensor-based mapping (LiDAR/RGB-D). These are often unavailable in real-world scenarios or add significant hardware cost and system complexity.
Scalability: Multi-robot cooperation often relies on inter-robot communication, which limits scalability and efficiency.

Goal: Develop a decentralized, end-to-end visuomotor policy that maps raw onboard camera images directly to robot velocity commands, eliminating the need for explicit state estimation, global localization, depth sensors, or inter-robot communication.

2. Methodology

The authors propose a Privileged Imitation Learning framework consisting of two stages: training a state-based "Teacher" and distilling its knowledge into a vision-based "Student."

A. Teacher Policy (Multi-Agent Reinforcement Learning - MARL)

Input: The teacher has access to "privileged" state information, including the relative positions of $K$ nearest obstacles, teammates, and enemies, plus the robot's own global state.
Architecture:
- Feature Extractor: Uses Self-Attention mechanisms to encode entities (obstacles, robots).
- Permutation-Invariance: A key design choice. Since the order of neighbors in a dynamic environment changes, the authors use weighted summation pooling on attention embeddings. This ensures the output is invariant to the input order of neighbors, preventing oscillatory behavior and ensuring robustness.
Training: Trained using MADDPG (Multi-Agent Deep Deterministic Policy Gradient) with a centralized training and decentralized execution paradigm.
Reward Function: Designed to encourage attacking enemies (distance and aiming rewards) while penalizing collisions with obstacles, boundaries, and teammates (cross-fire avoidance).

B. Student Policy (Vision-Based Imitation Learning)

Input: A time series of $N$ onboard monocular images.
Preprocessing & Input Representation:
- Object Detection: Uses YOLOv5 Nano to detect allies and enemies.
- Target Heatmaps: Instead of raw bounding boxes, the detected locations are converted into Gaussian heatmaps. The kernel size is inversely proportional to the distance (closer objects = larger kernels), emphasizing the aiming goal.
- Depth Estimation: Uses Depth Anything v2 (DATv2) to generate a monocular depth map. This provides appearance-invariant geometric cues without physical depth sensors.
- Input Tensor: Concatenates the RGB image (or just the heatmaps), the depth map, and the target heatmaps.
Architecture:
- Visual Encoder: A CNN (e.g., ResNet18 or SimpleCNN) processes the input tensor.
- Temporal Module: An LSTM aggregates the time-series of image embeddings to handle the limited Field of View (FOV) and temporal inconsistencies in depth estimation.
- Action Head: An MLP regresses the velocity command ( $v_x, v_y, \omega$ ).
Training: Uses Dataset Aggregation (DAgger) to collect data from the teacher policy in both simulation and real-world environments. The loss is the $L_2$ norm between the student's action and the teacher's action.

3. Key Contributions

Decentralized End-to-End Policy: A novel system that operates solely on onboard monocular vision and proprioception, removing dependencies on global localization, depth sensors, and inter-robot communication.
Performance Gains: The method outperforms classic modular baselines by 16.7% in hitting accuracy and 6% in collision avoidance.
Technical Innovations:
- Permutation-Invariant Feature Extractor: Ensures stable policy behavior in dynamic multi-robot environments by handling variable neighbor orders.
- Depth-Heatmap Input: Combines monocular depth estimation with Gaussian target heatmaps to create a robust, appearance-invariant input representation that improves aiming accuracy.
Real-World Deployment: Successfully deployed on a multi-robot system with limited onboard compute (Nvidia Jetson Orin NX), demonstrating practical viability.

4. Experimental Results

The authors evaluated the system in simulation and on real robots (4m $\times$ 4m arena with obstacles).

Ablation Studies:
- Input Types: The combination of Depth + Heatmaps significantly outperformed pure RGB, edge images, or depth-only inputs. The heatmap provides clear aiming cues, while depth provides geometric robustness.
- Feature Extractors: The Self-Attention with Pooling (SAP) design (Teacher-SAP) achieved the highest hit rates (48.6 $\pm$ 1.7 for Hit@15) compared to standard MLP or vanilla self-attention, confirming the importance of permutation invariance.
- Imitation Methods: The DAgger + LSTM approach for the student policy yielded the best results (38.4 Hit@15), proving that temporal aggregation is crucial for handling limited FOV.
Comparison with Baselines:
- Compared against classic pipelines (Bearing-Angle estimation + PD controller, Visual Servoing, etc.), the proposed Student Policy achieved a 38.4 Hit@15 score, significantly higher than the best modular baseline (33.0).
- The Teacher Policy (with privileged states) achieved 48.6 Hit@15, indicating that the primary bottleneck for the student is the lack of perfect state estimation, yet the student still outperforms all state-estimation-based modular methods.
- Collision Rate: The proposed method reduced collision rates by 6% compared to the strongest baseline.
Efficiency: The full pipeline (YOLOv5 + DATv2 + Policy) runs at 20 Hz on the Jetson Orin NX, with the visual encoder inference time under 20ms, ensuring smooth real-time control.

5. Significance

This work demonstrates that end-to-end visuomotor policies can effectively solve complex, dynamic multi-robot adversarial tasks without relying on fragile state estimation pipelines or expensive hardware. By mimicking human behavior (reacting directly to visual input rather than calculating explicit states), the system achieves:

Robustness: Better performance in scenarios where state estimation fails (e.g., occlusion, noise).
Scalability: No need for inter-robot communication or global tracking systems.
Practicality: Proven on low-cost, resource-constrained hardware, making it applicable to real-world tasks like drone interception, security patrols, and search-and-rescue operations where explicit state estimation is difficult.