Tiny-DroNeRF: Tiny Neural Radiance Fields aboard Federated Learning-enabled Nano-drones

This paper introduces Tiny-DroNeRF, a lightweight Neural Radiance Field model optimized for sub-100mW microcontrollers on nano-drones and enhanced by a federated learning scheme, which achieves dense 3D scene reconstruction with a 96% memory reduction and improved accuracy despite extreme resource constraints.

The Gist

Imagine you have a tiny, buzzing drone, no bigger than a hummingbird and weighing less than a smartphone. This little robot is designed to zip through tight, cluttered spaces—like the inside of a collapsed building or a messy factory—where big drones can't fit.

But there's a catch: because it's so small, its brain (its computer chip) is incredibly weak. It has the processing power of a basic calculator and very little memory, like a tiny notepad.

The Problem: The "Heavy" 3D Brain
Usually, to make a drone understand a 3D room (so it doesn't crash and can map the area), we use a super-smart AI called NeRF (Neural Radiance Fields). Think of NeRF as a magical artist that can look at a few photos and paint a perfect, photorealistic 3D movie of the scene.

However, this "magical artist" is a glutton. It usually requires a massive supercomputer (like a high-end gaming PC) with a huge hard drive and a power-hungry graphics card. Trying to run this on our tiny drone is like trying to fit a full-sized refrigerator into a backpack. It just doesn't fit, and the battery would die instantly.

The Solution: Tiny-DroNeRF
The authors of this paper created Tiny-DroNeRF. Think of this as shrinking that massive refrigerator down into a compact, energy-efficient lunchbox that fits perfectly in the drone's backpack.

Here is how they did it, using some creative tricks:

1. The "Lunchbox" Optimization: They took the standard, heavy NeRF recipe and stripped away the fat. They adjusted the settings (like how many photos to look at at once and how detailed the map needs to be) so the drone could handle it.

   • The Result: They cut the memory usage by 96%. It's like taking a 500-page book and condensing it into a 20-page pamphlet. The quality dropped a little bit (the picture isn't quite as sharp), but it's still good enough to see the walls and obstacles.

2. The "Teamwork" Strategy (Federated Learning): Even with the lunchbox, one tiny drone can only see a small part of a room. If it tries to learn the whole building alone, it runs out of memory before it's done.

   • The Analogy: Imagine a group of explorers in a dark cave. Each explorer has a flashlight and can only see a few feet ahead. If they try to draw a map of the entire cave alone, they will get lost.
   • The Fix: Instead of sharing their photos (which would clog their tiny radios), the drones share their learnings. Drone A learns about the left wall, Drone B learns about the ceiling, and Drone C learns about the floor. They meet up, swap their "mental notes" (the AI model updates), and combine them into one giant, shared map.
   • The Magic: This allows the swarm to build a complete, high-quality 3D map of a huge area, even though no single drone has enough brainpower or memory to do it alone.

Why This Matters
Before this, if you wanted a tiny drone to map a disaster zone or inspect a factory, it had to be "dumb" or rely on a human operator holding a cable.

With Tiny-DroNeRF:

• The drone is independent: It can fly into a dangerous place, build a 3D map of the inside, and figure out where to go without needing a supercomputer back at base.
• The swarm is smarter: A group of these tiny drones can work together to map a whole building faster and more accurately than any single drone could.
• It's efficient: It runs on a chip that uses less power than a nightlight (60 milliwatts), meaning the drone can fly for a long time without needing a recharge.

In a Nutshell
The researchers took a technology that usually requires a supercomputer, shrunk it down to fit in a toy drone's brain, and taught a swarm of these drones how to share their knowledge to build a perfect 3D map of the world around them. It's like turning a team of ants into a single, super-intelligent architect.

Technical Summary

1. Problem Statement

Nano-drones (sub-30g, sub-10cm) offer unique capabilities for exploring cluttered, narrow, and hazardous environments (e.g., industrial plants, disaster zones) due to their agility and autonomy. However, their extreme resource constraints—specifically sub-100 mW power budgets, <100 MB memory, and limited compute (~100 GOps/s)—render complex vision tasks like dense 3D scene reconstruction infeasible.

Current state-of-the-art (SoA) methods for 3D reconstruction, such as Neural Radiance Fields (NeRF), typically require:

• Gigabytes of memory (e.g., Instant-NGP requires ~527 MB).
• High-end GPUs consuming hundreds of Watts.
• Offline execution, preventing real-time onboard processing.

The challenge is to enable onboard, autonomous 3D reconstruction (specifically Novel View Synthesis) on these ultra-low-power (ULP) microcontroller units (MCUs) without relying on external infrastructure or heavy compute resources.

2. Methodology

The authors propose Tiny-DroNeRF, a solution combining a highly optimized NeRF algorithm with a collaborative Federated Learning (FL) framework.

A. Hardware-Aware Memory-Efficient NeRF (Tiny-DroNeRF)

Based on Instant-NGP, the authors developed a lightweight variant specifically optimized for the GAP9 Ultra-Low-Power (ULP) MCU (GreenWaves Technologies) aboard their nano-drone.

• Hyperparameter Optimization: They performed a rigorous search to find the trade-off between reconstruction quality (PSNR), memory footprint, and computation. Key optimizations included:
  • Reducing images per training step (from 100 to 1) to minimize off-chip memory access.
  • Lowering input image resolution (800×800 $\to$ 160×160).
  • Reducing batch size ($B$) and the maximum parameters per hash level ($T$) in the Multi-Resolution Hash Encoding (MRHE).
• Memory Management:
  • Tiling Strategy: Data is subdivided into tiles that fit within the 128 kB L1 memory to avoid stalling on memory transfers.
  • Batch Accumulation: To maintain convergence quality despite small batch sizes, gradients are accumulated over multiple steps before updating weights.
  • Memory Locality: The implementation ensures that most data resides in L1/L2, minimizing slow L3 (off-chip) transfers to only ~3.6 MB per step.
• Implementation: Custom handwritten C kernels utilizing GAP9's SIMD instructions and multi-core cluster for parallel execution.

B. Federated Learning Scheme

To overcome the limitation of a single drone capturing only a partial view of a scene, the authors implemented a Federated Learning (FL) approach across a swarm.

• Architecture: One drone acts as a coordinator (also training locally), while others act as clients.
• Process:
  1. Pre-training: The coordinator pre-trains on its local data.
  2. Local Training: All drones train locally for $N_\ell$ steps.
  3. Aggregation: Drones send model updates (Hash tables and MLP parameters) to the coordinator.
  4. Global Update: The coordinator aggregates updates using FedAvg (arithmetic mean) and broadcasts the global model.
• Optimization: The authors found that exchanging the Occupancy Grid (which accounts for ~88% of communication volume) was unnecessary for performance. Omitting it reduced communication overhead from 20.16s to 2.24s per round.

3. Key Contributions

1. Tiny-DroNeRF Algorithm: The first NeRF model tailored for ULP MCUs, achieving a 96% reduction in memory footprint compared to Instant-NGP (from 527 MB to 21.4 MB) with only a 5.7 dB drop in PSNR.
2. Embedded Implementation: A fully optimized deployment on the GAP9 MCU, achieving a full forward-backward step in 73 ms (97 minutes for full training on a single drone).
3. Federated Learning for Nano-drones: The first demonstration of FL on a swarm of resource-constrained nano-drones, enabling the synthesis of unseen scene portions by leveraging peer knowledge without exchanging raw images.
4. Real-World Validation: Successful reconstruction of real-world indoor scenes using onboard grayscale images, achieving 21.2 PSNR.

4. Experimental Results

• Performance vs. Instant-NGP:
  • Memory: Reduced from 527 MB to 21.4 MB (96% reduction).
  • Accuracy: PSNR dropped from 33.18 dB (Instant-NGP on GPU) to 26.94 dB (Tiny-DroNeRF on MCU) on the Synthetic NeRF dataset.
  • Speed: Training time per step is 73 ms; full training takes ~97 minutes on a single drone.
• Federated Learning Performance:
  • IID (Independent Identically Distributed) Data: Federated training achieved a PSNR 0.7 dB higher than the best single-drone model and only 0.6 dB below a centralized model trained on all data.
  • Non-IID (Partitioned Viewpoints): Federated models outperformed single-drone models by 1.7 dB, proving the swarm can infer geometry of unseen areas.
  • Communication Overhead: Only 2.24 seconds of communication required every 1,000 training steps.
• In-Field Deployment:
  • Tested on a real indoor scene with narrow walls and obstacles.
  • Achieved 21.2 PSNR and 0.73 SSIM using 160×160 grayscale images, successfully reconstructing corridor geometry and objects.

5. Significance

This work represents a paradigm shift in embedded robotics:

• Feasibility: It proves that complex 3D reconstruction tasks, previously restricted to high-power GPUs, can run autonomously on sub-100 mW nano-drones.
• Swarm Intelligence: It demonstrates that a swarm of tiny robots can collaboratively build a high-fidelity 3D map of an environment, overcoming the "blind spots" of individual agents through Federated Learning.
• Resource Efficiency: By combining algorithmic compression (Tiny-DroNeRF) with distributed learning, the system maximizes the utility of extremely limited hardware, enabling new applications in search and rescue, industrial inspection, and autonomous navigation in GPS-denied environments.

This is the first demonstration of on-device NeRF training on a highly constrained MCU combined with a federated learning scheme on a nano-drone swarm.