🤖 machine learning

SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations

SWAN is a novel adaptive multimodal network designed for autonomous driving that dynamically allocates resources across modalities and scales layer utilization based on input complexity and user-defined budgets, achieving up to a 49% reduction in FLOPs with minimal performance degradation.

Original authors: Jason Wu, Shir-Kang Scott Jinn, Yuyang Yuan, Maggie Wigness, Lance M. Kaplan, Hang Qiu, Mani Srivastava

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jason Wu, Shir-Kang Scott Jinn, Yuyang Yuan, Maggie Wigness, Lance M. Kaplan, Hang Qiu, Mani Srivastava

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a self-driving car. Usually, the car's "brain" (its AI) is designed to run at full power, using all its sensors and processing power to see the road. But in the real world, things change constantly:

The sensors get dirty: It might start raining, or fog might roll in, making the camera blind while the laser scanner (LiDAR) still works fine. Or maybe the battery is low, and the car can't afford to run all its heavy software.
The road gets busy or empty: Sometimes there are hundreds of cars and pedestrians (a complex scene), and sometimes it's just an empty highway (a simple scene).

Current self-driving AI is like a stubborn chef who insists on using the same massive, expensive recipe whether they are cooking for one person or a hundred, or whether the ingredients are fresh or rotten. It wastes energy on easy tasks and struggles when conditions get tough.

The paper introduces SWAN (Sample and World-Aware Multimodal Network), a new type of AI that acts like a smart, adaptive project manager for the car's brain.

Here is how SWAN works, broken down into three simple tricks:

1. The "Quality-Aware Manager" (The Controller)

Imagine you have a fixed budget for a construction project, but the quality of your bricks (the data from sensors) keeps changing.

The Problem: If the camera is covered in mud (bad quality), but the laser scanner is clear, a normal AI might try to use both equally, wasting money on the muddy camera.
SWAN's Solution: SWAN has a manager that constantly checks the "quality" of the data. If the camera is foggy, the manager says, "Stop spending budget on the camera! Give all our money to the laser scanner instead." It dynamically shifts resources to the sensor that is working best at that exact moment, ensuring the car doesn't crash just because one sensor is having a bad day.

2. The "SkipGate" (The Smart Skip)

Imagine you are reading a book. If you are reading a simple children's story, you might skim through the easy pages. But if you hit a complex chapter, you read every single word carefully.

The Problem: Current AI reads every single page of the "book" (runs every layer of the neural network) even if the scene is simple.
SWAN's Solution: SWAN has a "SkipGate" that looks at the current scene. If the road is empty and the weather is clear, it says, "We don't need to process this whole section; let's skip a few layers to save energy." It only does the heavy lifting when the scene is actually complicated. This saves a huge amount of battery and processing power.

3. The "Token Pruner" (The Trash Collector)

Imagine you are looking at a photo of a forest. You need to find a deer. The photo has thousands of pixels, but 90% of them are just blue sky or green leaves that don't matter.

The Problem: The AI tries to analyze every single pixel, even the boring background.
SWAN's Solution: Before the AI tries to find the deer, SWAN acts like a trash collector. It quickly scans the image and throws away the "useless" pixels (like the sky or distant trees) that don't help find the car or pedestrian. It focuses all its energy only on the interesting parts of the image.

The Results: A Smarter, Faster Car

The researchers tested SWAN in a simulated world with rainy, dark, and foggy conditions, and even on real hardware (a small computer chip used in robots).

Efficiency: SWAN managed to cut the computer work (FLOPs) by up to 49% (almost half!) without losing much accuracy.
Robustness: When the sensors were "corrupted" (like heavy fog or motion blur), SWAN performed much better than standard systems because it knew exactly which sensor to trust and how to adjust.
Real-World Speed: On a small, low-power computer chip (like the kind in a real robot), SWAN was significantly faster and used less power than the old methods.

In short: SWAN is a self-driving AI that knows when to work hard, when to take a break, and which sensors to trust, making it safer and more energy-efficient in the messy, unpredictable real world.

1. Problem Statement

Multimodal deep neural networks deployed in real-world environments, particularly autonomous vehicles (AVs), face three critical runtime variations that current systems struggle to handle simultaneously:

Modality Quality of Information (QoI): Environmental changes (e.g., fog, darkness, sensor failure) degrade the quality of specific sensor inputs (camera vs. LiDAR).
Platform Dynamics: Available computational resources fluctuate due to thermal throttling, power constraints, or battery levels.
Sample Complexity: The difficulty of the input varies (e.g., a clear road with few objects vs. a complex intersection with many targets).

Limitations of Existing Approaches:

Static Multimodal Networks: Fail to adapt to QoI changes or resource constraints, leading to wasted computation or performance drops.
Budget-Adaptive Networks: Can adjust to compute limits but often ignore input complexity or modality quality.
Input-Adaptive Networks: Optimize for sample complexity but often lack strict compute budget guarantees or fail to handle multimodal redundancy.

The core challenge is the compounding complexity of jointly addressing these three dynamics while maintaining end-to-end differentiability for training.

2. Methodology: SWAN Architecture

SWAN (Sample and World-Aware Multimodal Network) is built upon the CMT (Camera-LiDAR Multimodal Transformer) framework. It introduces three novel components to achieve adaptive resource allocation:

A. QoI-Aware Controller

Function: Dynamically allocates a user-specified maximum compute budget (in terms of layer count) across different modality backbones (Camera and LiDAR) based on the current input's QoI.
Mechanism:
- Extracts QoI features from inputs using lightweight convolutional networks.
- Concatenates QoI features with a positional embedding representing the current budget.
- Outputs logits for every layer in every modality.
Training Innovation: To handle the non-differentiable nature of selecting specific layers, SWAN uses NeuralSort, a differentiable sorting algorithm. This allows the controller to learn an optimal ranking of layers during training, approximating the discrete selection process required at inference. This overcomes the convergence issues found in previous "Straight-Through" gradient methods.

B. SkipGate Module (Feature-Aware Layer Dropping)

Function: Operates within the budget allocated by the controller to further reduce computation on "easy" samples. It conditionally executes layers based on feature complexity.
Mechanism:
- Takes contextual inputs: current layer index, remaining budget, layers assigned to the other modality, and the controller's QoI embedding.
- Uses a Gumbel-Sigmoid operator to generate a probability of executing the current layer.
- Includes a utilization penalty loss ( $L_{skip}$ ) to encourage the model to skip layers whenever possible without sacrificing accuracy.

C. Token Pruning Module

Function: Reduces computational load at the detection head by filtering out semantically irrelevant tokens (e.g., sky, background trees) before the final detection head.
Mechanism: A lightweight convolutional network predicts a weight map for tokens. A rounding operation (with straight-through estimation) discretizes these weights to drop tokens. An additional utilization loss ensures only necessary tokens are retained.

D. Joint Training Strategy

The model is trained in a staged manner:

Train the base network with LayerDrop.
Freeze the backbone and train the QoI Controller.
Freeze the controller and backbone, then train SkipGate modules.
Finally, train the Token Pruning module.
This staged approach ensures stability while integrating the adaptive components.

3. Key Contributions

First Unified Adaptive Multimodal Network: SWAN is the first system to jointly address modality QoI, strict compute budgets, and sample complexity in a multimodal setting.
Differentiable Controller via NeuralSort: Introduces a QoI-aware controller that uses NeuralSort for end-to-end differentiable layer allocation, outperforming previous Straight-Through methods in complex AV scenarios.
Hierarchical Efficiency: Combines a global budget allocator (Controller) with local efficiency modules (SkipGate and Token Pruning) to maximize utility within constraints.
Comprehensive Evaluation: Validated on synthetic corrupted data (MultiCorrupt pipeline) and real-world edge hardware (Nvidia Jetson Orin), demonstrating robustness and practical efficiency.

4. Experimental Results

The authors evaluated SWAN on the nuScenes dataset with simulated corruptions (LiDAR beam reduction, fog, darkness, motion blur) and compared it against baselines like BEVFusion, SST, PETR, and the adaptive ADMN.

Performance under Budget Constraints:
- Under tight budgets (e.g., 4 layers), SWAN significantly outperforms baselines. For example, with LiDAR beam reduction, SWAN improved mAP from 3.88 (Naive baseline) to 16.12.
- SWAN consistently outperforms ADMN, which struggles with precise layer selection due to its training methodology.
Efficiency Gains:
- FLOPs: Reduced by up to 49% compared to a fully-provisioned network.
- Latency: On edge devices (Jetson Orin), SWAN reduced latency by >50% with token pruning and up to 16.5% with SkipGate under power constraints.
- Accuracy: SWAN achieved comparable accuracy to a full-network baseline while using only a fraction of the resources (e.g., within 2.7 NDS and 5.4 mAP of the upper bound with only 4 layers).
Real-World Validation:
- SWAN demonstrated strong generalization to real-world rainy and dark data, even when trained primarily on synthetic corruptions.
- Fine-tuning the SkipGate module on real data further recovered performance in high-budget scenarios where aggressive dropping initially hurt accuracy.

5. Significance and Impact

Safety and Robustness: By dynamically prioritizing high-quality modalities (e.g., relying on LiDAR in darkness and cameras in fog), SWAN enhances the safety of autonomous systems in unpredictable environments.
Resource Efficiency: The ability to scale computation based on both hardware limits and input complexity allows for deployment on a wider range of edge devices (from mobile robots to high-end AVs) without sacrificing performance.
Methodological Advancement: The successful application of NeuralSort for multimodal resource allocation provides a new blueprint for training adaptive networks where discrete decisions must be made differentially.
Practical Deployment: The demonstration on Nvidia Jetson Orin proves that the theoretical gains in FLOPs translate to tangible latency reductions in real-world, resource-constrained hardware, addressing the "orchestration overhead" often seen in high-performance GPU testing.

In conclusion, SWAN represents a significant step forward in making multimodal AI systems truly adaptive, robust, and efficient for the dynamic realities of the physical world.