💻 computer science

G-MAD: A Game-Based Data Generation Framework for Multi-View RGB-T Aerial Object Detection

This paper introduces G-MAD, an open-source framework leveraging Arma3 to generate synchronized multi-view RGB-T aerial data with automatic annotations, and utilizes it to release AMOD, a new large-scale benchmark for advancing aerial object detection research.

Original authors: Yechan Kim, JongHyun Park, Dongho Yoon, Namhoon Jung, Moongu Jeon

Published 2026-07-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yechan Kim, JongHyun Park, Dongho Yoon, Namhoon Jung, Moongu Jeon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to spot things from the sky, like a drone looking for a lost hiker or a military vehicle. This is a tricky job because the robot has to deal with the world from a very strange angle: looking straight down or from a steep side, where things look different than they do on the ground. To teach a robot, you usually need thousands of photos with labels saying "that's a tank" or "that's a tree." But taking real photos from the sky is expensive, dangerous, and hard to control. You can't just ask a drone to fly over a specific spot at exactly 2:00 PM on a rainy Tuesday while holding a thermal camera (which sees heat instead of light) and a regular camera at the same time.

This is where "synthetic data" comes in. Think of it like a video game. Instead of going out into the real world to take photos, scientists build a digital world inside a computer. They can drop a virtual tank into a virtual forest, change the weather to a thunderstorm, and take a picture instantly. The computer knows exactly where the tank is, so it can write the label for the photo automatically. This paper dives into a specific corner of this world: using a military simulation game to create a massive library of training photos for robots that need to see in both visible light and heat.

The researchers behind this paper, Yechan Kim and his team, noticed that while video games are great for making pictures, most of them (like the famous Grand Theft Auto) are built for cars and people on the ground. They don't have the special "thermal" vision needed to see heat signatures, and they aren't set up to take pictures from a drone's perspective. So, the team built a new tool called G-MAD. They used a different game, Arma 3, which is famous for its realistic military simulations and open-world maps.

Think of G-MAD as a super-powered, automated camera crew living inside the game. Instead of a human player flying a drone and taking pictures one by one, G-MAD is a script that says: "Okay, drop 50 tanks and 20 trucks in this forest. Make it 2:00 PM on a cloudy day. Now, fly a virtual drone over them, taking pictures from every possible angle, and make sure you take a regular photo and a heat-photo at the exact same time." The magic part is that because the computer controls the game, it knows the exact 3D shape and position of every single object. It doesn't need a human to draw boxes around the tanks; the computer calculates the perfect outline and saves it instantly. This solves a huge problem: getting perfectly matched pairs of regular and heat images is incredibly hard in the real world, but in this game, it's automatic.

Using this tool, the team created a new dataset they call AMOD. It's like a giant, organized photo album containing nearly 74,000 images. These aren't just random snapshots; they are carefully organized so that for every single scene (like a specific group of vehicles in a field), the system took pictures from many different angles and with both types of cameras. The dataset includes 12 different categories of military objects, from armored vehicles and tanks to helicopters and radar systems.

The team tested if this game-made data actually helps real robots learn. They taught a computer model using these game photos and then asked it to find things in real-world pictures. The results were promising. When the model was trained only on pictures from one angle, it got confused when the angle changed. But when it was trained on the multi-angle photos from G-MAD, it became much better at spotting objects from any direction. Furthermore, when they used the game data to give the robot a "head start" before teaching it real-world photos, the robot performed better than if it had started from scratch. The team even showed that a robot trained on their game data could spot military vehicles in real news photos from the Russia-Ukraine conflict, even though it had never seen those specific real-world images before.

The paper suggests that this approach is a powerful way to build training data without the cost and danger of real flights. However, the authors are careful to note that their tool is specialized for military targets because the game they used is built for war simulations. While the system is flexible enough to add civilian objects, the current dataset is focused on tanks, planes, and artillery. They also emphasize that this is a simulation-based study; the "proof" is in how well the computer models perform in tests, not in a real-life deployment of a drone. Ultimately, G-MAD offers a playful but serious way to generate the massive amounts of data needed to teach machines how to see the world from above, bridging the gap between a video game and real-world safety.

Technical Summary: G-MAD – A Game-Based Data Generation Framework for Multi-View RGB-T Aerial Object Detection

1. Problem Statement

Aerial object detection faces significant challenges due to large viewpoint variations, scale changes, and complex backgrounds. These difficulties are exacerbated in RGB-Thermal (RGB-T) settings, where models must leverage complementary visible and thermal cues while maintaining accurate cross-modal correspondence. Constructing real-world aerial RGB-T datasets is prohibitively expensive and labor-intensive, requiring synchronized sensor acquisition, controlled flight conditions, and manual annotation. Existing datasets often lack control over viewpoints, environmental factors, and precise modality alignment. While synthetic data generation offers a solution, existing game-based pipelines (e.g., those using GTA V) often lack native thermal support or structured controllability for multi-view aerial scenarios. Furthermore, previous Arma3-based frameworks suffer from limitations such as single-object constraints, manual camera setups, and reliance on error-prone image differencing for annotation.

2. Methodology: G-MAD Framework

The authors propose G-MAD (a data Generator for Multi-view Aerial object Detection using Arma3), an open-source framework that transforms the military simulation game Arma3 into a structured, end-to-end data generation pipeline. The system is implemented in SQF (Arma3's native scripting language) and operates through three primary stages:

2.1 Scenario Specification

Instead of manual construction, G-MAD generates scenarios automatically based on user-defined high-level constraints. These constraints include:

Asset Categories: Selection from a pre-defined mapping of 513 Arma3 models to 12 object classes.
Object Density: Expected minimum and maximum numbers of objects to spawn.
Environment: Weather, time of day (day/night), and map selection.
Sensor Configuration: Altitude, Field of View (FOV), modality (RGB/Thermal), image resolution, and camera view sampling strategy.
Scale: The total number of scenarios to generate.

2.2 Scenario Sampling and Multi-View, Multi-Modal Capture

The framework decouples scenario-level variables (object arrangement, environment) from scene-level variables (camera pose, sensing modality).

Sampling Process: A single scenario ( $s$ ) is sampled from the constraint space, and multiple scenes ( $v$ ) are generated by varying viewpoints and modalities while keeping the underlying object layout fixed.
Viewpoint Strategies: G-MAD supports user-defined sampling models. Default modes include:
- "Air-to-Air": Lateral movement at stable altitudes for sweeping aerial observations.
- "Air-to-Land": Parabolic trajectories with varying altitude and lateral position for oblique ground views.
Synchronization: The system captures synchronized RGB and thermal images simultaneously, ensuring pixel-perfect alignment and identical annotations across modalities.

2.3 Automatic Annotation and Export

Unlike prior methods relying on foreground-background image differencing, G-MAD queries engine-level geometric metadata directly from the Arma3 engine.

Projection: 3D bounding box vertices of objects are projected onto the 2D image plane using camera parameters.
Output: The system generates both Horizontal Bounding Boxes (HBB) and Oriented Bounding Boxes (OBB).
Filtering: An automatic occlusion check filters out invalid annotations (e.g., objects fully occluded with no direct line-of-sight), reducing label noise.

3. Key Contributions

G-MAD Framework: A novel, open-source pipeline that enables structured, constraint-aware, multi-object, multi-view, and multi-modal (RGB-T) data generation using Arma3. It supports GUI-independent execution, supersampling for higher resolution, and automatic cache cleanup.
AMOD Dataset: The construction and release of AMOD (Aerial Military Object Detection), a large-scale benchmark containing 73,920 frames with 383,200 annotated instances across 12 military categories. The dataset features synchronized RGB-T pairs, OBB annotations, and multiple viewpoints per scenario.
Comparative Advantages: The framework addresses limitations of existing Arma3 tools by supporting multiple objects per image, configurable camera sampling models, and engine-native ground truth, eliminating the need for brittle GUI automation or image differencing.

4. Experimental Results

The authors conducted benchmarking experiments to validate the utility of G-MAD and the AMOD dataset using an Oriented R-CNN with a Swin-S backbone.

Multi-View Learning: Training on multi-view data significantly improved generalization to unseen angles compared to single-view training. The multi-view model achieved a mean AP50:95 of 50.96, outperforming the best single-view model (44.57) by +6.39.
Synthetic-to-Real Transfer: Pretraining on AMOD (both RGB and thermal variants) and fine-tuning on real-world benchmarks yielded consistent improvements:
- DIOR-R (Visible): Mean AP50 increased from 63.14 (ImageNet init) to 69.83 (+6.69).
- HIT-UAV (Thermal): Mean AP50 increased from 74.57 to 77.08 (+2.51).
Unseen Real-World Inference: A detector trained solely on AMOD (RGB split) was tested on unseen real-world satellite imagery of military equipment from public news coverage. The model successfully identified plausible military targets without fine-tuning, demonstrating the transferability of the synthetic data.

5. Significance and Claims

The paper positions G-MAD as a practical solution to the high cost and low controllability of real-world aerial RGB-T dataset construction. By leveraging the rich asset library and scriptable nature of Arma3, the framework provides a scalable way to study:

Viewpoint Robustness: Through controlled multi-view sampling.
Cross-Modal Fusion: Through synchronized, perfectly aligned RGB-T pairs.
Synthetic-to-Real Transfer: Providing a pretraining foundation for real-world aerial detection tasks.

The authors emphasize that while AMOD is specialized for military targets due to the source game's assets, the framework itself is extensible to non-military equipment via user-defined asset integration. The work is released for non-commercial academic research, with strict adherence to licensing regarding the underlying game engine.