TiROD: Tiny Robotics Dataset and Benchmark for Continual Object Detection

1. Problem Statement

The paper addresses the critical challenge of deploying Continual Object Detection (CLOD) systems on Tiny Robotics platforms. While object detection is vital for autonomous navigation and inspection, tiny robots face severe constraints:

• Hardware Limitations: Limited computational power (microcontrollers/mobile CPUs), memory, and energy.
• Sensor Quality: Use of low-cost cameras (e.g., ESP32-CAM) resulting in low-resolution, noisy images, motion blur, and poor signal-to-noise ratios.
• Dynamic Environments: Robots must operate in unstructured, changing environments (indoor/outdoor) with varying lighting and weather, leading to significant domain shifts.
• Catastrophic Forgetting: Standard neural networks lose previously learned knowledge when trained on new data. Existing CLOD benchmarks (e.g., CORe50, OpenLORIS) often fail to capture the specific combination of low-cost sensor noise, extreme resource constraints, and complex domain/class shifts found in real-world tiny robotics.

2. Methodology

A. The TiROD Dataset

The authors introduced TiROD (Tiny Robotics Object Detection), a new video dataset collected using a custom tiny mobile robot (SunFounder GalaxyRVR) equipped with an ESP32-CAM.

• Collection: Data was gathered across 5 distinct environments (1 indoor, 4 outdoor) with progressively rougher terrains (tiled, asphalt, grass, dirt).
• Conditions: Captured under two illumination levels ("High" and "Low") and augmented with synthetic weather conditions (rain, fog) for specific scenarios.
• Content: Contains 13 object classes (e.g., chair, bottle, potted plant, gardening tools) with a total of ~17.4k images.
• Annotation: Generated via Grounding Dino and manually refined to handle low-resolution/noisy challenges.
• Scenarios Defined:
  1. Cross-Domain Scenario (10 Tasks): Combines Domain-Incremental Learning (DIL) and Class-Incremental Learning (CIL). Tasks involve shifting between environments and introducing new object classes sequentially.
  2. Intra-Domain Scenario (4 Tasks): Focuses on appearance shifts (lighting, weather) within a single environment (Domain 4) without introducing new classes.

B. Benchmark Setup

The authors evaluated CLOD strategies using NanoDet-plus-m, a lightweight, real-time object detector (1.2M parameters, 0.9 Giga-FLOPs) suitable for edge devices.

• Baselines:
  • Lower Bound: Fine-Tuning (no memory, suffers catastrophic forgetting).
  • Upper Bound: Cumulative Training (retrained on all past data, no forgetting).
• Strategies Tested:
  • Regularization-based: Learning without Forgetting (LwF), Selective and Inter-related Distillation (SID), IncDet.
  • Replay-based: Standard Replay (random sampling), Temporal Replay (time-distant sampling), K-Means Replay (clustering feature vectors to select representative samples).
  • Latent Variants: Latent Distillation and Latent Replay (using frozen backbones for efficiency).
• Metrics:
  • Final mAP: Performance on all tasks after the final task.
  • RSD (Stability): Ability to retain performance on old tasks.
  • RPD (Plasticity): Ability to learn new tasks.

3. Key Contributions

1. TiROD Dataset: A challenging, real-world dataset specifically designed for tiny robotics, featuring low-cost sensor noise, motion blur, and diverse unstructured environments. It is the first to combine DIL and CIL scenarios tailored for resource-constrained platforms.
2. Comprehensive Benchmark: A systematic evaluation of CLOD strategies on NanoDet, establishing a reference for lightweight continual learning.
3. Novel Sampling Strategy: Introduction and evaluation of K-Means Replay, which clusters feature vectors to select the most representative samples for the memory buffer, outperforming random and temporal sampling.
4. Public Release: The dataset, code, and trained models are made publicly available to foster research in TinyML and robotics.

4. Results

Cross-Domain Scenario (Most Challenging)

• Replay vs. Regularization: Replay-based methods significantly outperformed regularization-based methods.
  • K-Means Replay achieved the best performance with a Final mAP of 42.5% and high stability (RSD 0.77).
  • Regularization (SID) achieved only 16.4% mAP, indicating that distillation alone is insufficient for severe domain shifts.
  • Fine-Tuning performed poorly (10.7% mAP) due to catastrophic forgetting.
• Sampling Impact: K-Means Replay (42.5%) outperformed Standard Replay (37.8%) and Temporal Replay (25.9%), proving that selecting diverse, representative samples is crucial when memory is limited.
• Robustness: Results remained consistent across reverse and randomized task orders.
• Buffer Size: Performance degraded as buffer size decreased (e.g., 21.9% mAP with a buffer of 25), highlighting the need for efficient memory usage.

Intra-Domain Scenario

• Performance gaps were smaller compared to Cross-Domain, as the class set remained constant.
• Replay methods still led (K-Means Replay: 66.0% mAP), but regularization methods (SID: 49.2%) performed relatively better than in the Cross-Domain setting.
• Fine-Tuning showed higher stability here (47.0% mAP), suggesting that appearance shifts are easier to adapt to than new class/domain combinations.

Backbone Analysis

• Testing larger backbones (MobileNetV2, EfficientNet-Lite) showed that increasing model size did not significantly close the performance gap with Cumulative Training. This confirms that TiROD is a challenging benchmark where the learning strategy (Replay) is more critical than model capacity.

5. Significance and Conclusion

The paper demonstrates that Replay-based strategies, particularly those using intelligent sampling like K-Means, are currently the most viable approach for Continual Object Detection on tiny robots. They effectively mitigate catastrophic forgetting in the face of severe domain shifts and noisy sensor data.

However, a significant performance gap remains between the best CLOD methods (42% mAP) and Cumulative Training (63% mAP). This gap highlights that current techniques are not yet optimal for real-world deployment where memory buffers must be extremely small. The work establishes TiROD as a necessary standard for evaluating future algorithms that aim to enable robust, lifelong learning on resource-constrained autonomous systems.