💻 computer science

CoInfra: A Large-Scale Cooperative Infrastructure Perception System and Dataset for Vehicle-Infrastructure Cooperation in Adverse Weather

This paper introduces CoInfra, a large-scale cooperative infrastructure perception system and dataset featuring 14 roadside nodes connected via 5G under diverse adverse weather conditions, demonstrating that vehicle-infrastructure cooperation significantly enhances situational awareness and safety-critical detection in complex traffic scenarios compared to vehicle-only sensing.

Original authors: Minghao Ning, Yufeng Yang, Keqi Shu, Shucheng Huang, Jiaming Zhong, Maryam Salehi, Mahdi Rahmani, Jiaming Guo, Yukun Lu, Chen Sun, Aladdin Saleh, Ehsan Hashemi, Amir Khajepour

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Minghao Ning, Yufeng Yang, Keqi Shu, Shucheng Huang, Jiaming Zhong, Maryam Salehi, Mahdi Rahmani, Jiaming Guo, Yukun Lu, Chen Sun, Aladdin Saleh, Ehsan Hashemi, Amir Khajepour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a self-driving car. Right now, these cars are like people wearing blindfolds who can only see what's directly in front of them. If a big truck blocks your view, or if a child runs out from behind a parked car, or if a heavy snowstorm makes it hard to see, the car might not know the danger is there until it's too late.

This paper introduces CoInfra, a solution that gives self-driving cars "super-vision" by connecting them to a network of smart cameras and sensors mounted on streetlights and poles.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Blind Spot" of the Self-Driving Car

Think of a self-driving car as a blindfolded chess player. They can only see the pieces immediately around them.

The Issue: In a busy roundabout (a circular intersection), a car might be hidden behind a bus. In bad weather (like freezing rain or heavy snow), the car's "eyes" (cameras and lasers) get blurry or confused.
The Result: The car makes decisions based on incomplete information, which is dangerous.

2. The Solution: The "Hive Mind" of Streetlights

CoInfra is like installing security cameras on every streetlight around the roundabout.

The Setup: The researchers set up 14 of these "smart poles" around a real roundabout in Canada. Each pole has its own laser scanner (LiDAR) and cameras.
The Connection: These poles talk to each other and to the car instantly using 5G (the same super-fast internet used on our phones, but much more reliable).
The Magic: Even if the car can't see a pedestrian because a truck is blocking the view, the "smart pole" on the other side of the road can see them. It tells the car, "Hey, there's a person there!"

3. The Big Challenge: The "Noisy Phone Call"

You might think, "Just send all the video feeds to the cloud!" But that's like trying to have a conversation with 14 people at once over a bad phone line.

The Bottleneck: Sending huge video files over 5G takes too long and often gets dropped (like a call cutting out).
The Fix: Instead of sending video, the poles send tiny, smart summaries. Imagine instead of sending a 2-hour movie, the pole just sends a text message saying: "I see a red car at 10 o'clock, moving fast." This is so small and fast that it arrives almost instantly, even in a snowstorm.

4. The "Traffic Controller" (Synchronization)

Since the poles are all sending messages at slightly different times (some messages arrive faster than others), the system needs a Traffic Controller.

The Problem: If the car waits for every single message to arrive before making a decision, it might wait too long and cause an accident.
The Solution: The system uses a clever "wait-and-see" strategy. It waits just long enough to get most of the messages, and if a few are late, it uses a GPS-like prediction to guess where those missing objects are. This keeps the car moving smoothly without waiting for a perfect signal.

5. The Results: From "Maybe" to "Definitely"

The researchers tested this system in four weather conditions: sunny, rainy, snowy, and freezing rain.

The Old Way (Car alone): In the most dangerous situations (like a pedestrian stepping out while the car is in the middle of the roundabout), the car could only "see" the danger 33% to 46% of the time. It was often blind.
The New Way (Car + Smart Poles): With the help of the poles, the car could "see" the danger 86% to 100% of the time.
The Analogy: It's like the difference between trying to find a needle in a haystack while wearing sunglasses (the car alone) versus having a team of friends holding flashlights who can see the needle from every angle (CoInfra).

6. Why This Matters

This isn't just a computer simulation; it's a real-world system that actually works.

It's Open Source: The team didn't just keep the secret; they released the blueprints, the code, and the data so other engineers can build it too.
It's Weather-Proof: It works even when it's snowing or raining hard, conditions where current self-driving cars struggle the most.

The Bottom Line

CoInfra proves that self-driving cars don't have to be lonely. By connecting them to a network of "smart eyes" on the street, we can make them safer, especially in the tricky, dangerous moments where a single car's view is blocked. It turns a solo driver into a member of a team that never misses a thing.

1. Problem Statement

Autonomous driving perception faces significant limitations in complex urban environments, particularly regarding occlusions, large-scale coverage, and adverse weather conditions (rain, snow, freezing rain). While Vehicle-to-Infrastructure (V2I) cooperation offers a solution by providing elevated, complementary viewpoints, current research and deployment face three critical gaps:

Synchronization Neglect: Existing systems often treat multi-node synchronization and stochastic communication delays as implementation details rather than explicit research objects.
Lack of Reproducible Full-Stacks: Most prior work releases only datasets or algorithms, lacking the integrated hardware, calibration, and software tools needed to replicate large-scale deployments.
Evaluation Limitations: Benchmarks rely heavily on detector-level metrics (e.g., Average Precision) and often fail to capture adverse weather diversity or evaluate whether cooperation actually improves situational awareness in safety-critical conflict scenarios (e.g., roundabouts).

2. Methodology and System Design

A. The CoInfra Platform

The authors developed a deployable cooperative infrastructure perception system comprising 14 roadside sensor nodes connected via a commercial 5G network.

Hardware: Each node includes a 32-beam LiDAR, two RGB cameras, an NVIDIA Jetson Orin NX edge computer, and a 5G modem.
Deployment: Nodes are mounted on light poles (~7.5m height) around a complex urban roundabout at the University of Waterloo, chosen for its high frequency of merging, yielding, and vulnerable road user (VRU) interactions.
Pipeline: Nodes perform local perception (LiDAR-camera fusion) at the edge and transmit structured outputs (bounding boxes/tracks) rather than raw sensor data to minimize latency.

B. Delay-Aware Synchronization

To handle stochastic 5G latency, the system employs a probabilistic fusion-window policy:

Latency Modeling: The system models per-node delay as a Gaussian distribution ( $\delta \sim N(\hat{\mu}, \hat{\sigma}^2)$ ) based on real-time measurements.
Adaptive Window: A fusion window $\Delta T$ is calculated as $\max_i(\hat{\mu}_i + n\hat{\sigma}_i)$ (where $n=2$ or $3$). The cloud waits this duration before fusing data.
Post-Fusion Update: Messages arriving after the trigger are not discarded; they are integrated via a motion-model-based update (using Extended Kalman Filter predictions) with reduced confidence to maintain temporal consistency without delaying the primary output.

C. Dataset Construction

The CoInfra Dataset is organized into two subsets:

CoInfra-I2I (Infrastructure-to-Infrastructure): The primary benchmark containing 294k LiDAR frames and 589k camera images from 8 nodes under four weather conditions (Sunny, Rainy, Heavy Snow, Freezing Rain). It includes 332k globally consistent 3D bounding boxes.
CoInfra-V2I (Vehicle-to-Infrastructure): A synchronized subset collected with an autonomous shuttle, enabling evaluation of the full cooperation loop.

Annotation: Objects are annotated in a shared global coordinate frame using a Bird's-Eye-View (BEV) workflow, ensuring geometric consistency across nodes.

3. Key Contributions

Deployable System & Open-Source Stack: The release of a full-stack system including hardware documentation, calibration tools, a fleet management UI, and an Autoware-compatible ROS perception pipeline, enabling real-world replication.
Delay-Aware Synchronization Framework: A novel approach to multi-node synchronization that explicitly models 5G latency trade-offs, utilizing adaptive fusion windows and state prediction to minimize tail latency while preserving completeness.
Adverse-Weather Large-Scale Dataset: The first large-scale, multi-node V2I dataset covering diverse adverse weather conditions with globally consistent 3D annotations.
Transportation-Facing Evaluation: A shift from standard detector metrics to ego-centric observability analysis, quantifying how infrastructure sensing restores awareness of safety-critical participants in conflict zones.

4. Experimental Results

A. Communication Feasibility

Payload Analysis: Transmitting raw images over 5G resulted in high latency (>2s) and 63% packet loss. In contrast, compact object-level outputs (1–2 KB) achieved a mean latency of 24.2 ms with 0% packet loss, validating the edge-perception architecture.
Weather Impact: Adverse weather slightly increased tail latency (P99 increased from ~50ms to ~54ms) but did not significantly affect mean latency.

B. Synchronization Performance

Trace-Driven Evaluation: Using recorded 5G delays, an Adaptive 2 $\sigma$ strategy with EKF prediction reduced the P99 reaction time from 75.7 ms (Wait-for-all) to 57.0 ms while maintaining 100% object completeness. This demonstrates that adaptive synchronization effectively suppresses tail latency without sacrificing data integrity.

C. Perception Benchmarks (CoInfra-I2I)

Global vs. Local Fusion: Global fusion (early fusion in BEV) significantly outperformed local fusion (late fusion), particularly for VRUs (Pedestrians/Bicycles) and at longer distances.
Weather Robustness: In heavy snow, global LiDAR-camera fusion maintained high accuracy (AP@0.5 $\approx$ 0.585 for VRUs), whereas local fusion dropped significantly.
Point Density: Cooperative sensing increased the average LiDAR points per object from 11 (single node) to 20 for pedestrians and 23 to 60 for bicycles, crucial for detecting sparse objects.

D. Ego-Centric Observability (CoInfra-V2I)

This is the paper's most significant finding. The study measured "Critical-Frame Completeness" (the fraction of frames where all relevant conflict participants are visible).

Vehicle-Only Limitations: In roundabout conflict scenarios, vehicle-only sensing achieved only 33%–46% critical-frame completeness due to occlusions.
V2I Improvement: With cooperative infrastructure sensing, completeness rose to 86%–100%.
Specific Gains:
- AV in Ring $\to$ VRU Waiting: Visibility improved from 45.2% to 90.1%; Completeness from 33.0% to 85.9%.
- AV in Ring $\to$ Vehicle Entering: Visibility improved from 49.3% to 100.0%; Completeness from 46.1% to 100.0%.

5. Significance and Conclusion

The paper demonstrates that multi-node infrastructure perception is not just a theoretical improvement in detection accuracy but a critical enabler for safety-relevant situational awareness.

Practicality: It proves that cooperative perception is feasible under real-world 5G constraints using compact, object-level communication.
Safety Impact: It quantifies that infrastructure cooperation can restore visibility of occluded, safety-critical agents (pedestrians, other vehicles) in conflict zones where onboard sensors fail, potentially preventing accidents.
Community Impact: By releasing the full system stack and dataset, CoInfra lowers the barrier for researchers to study and deploy cooperative perception systems under realistic, adverse conditions, moving the field beyond simulation and idealized benchmarks.

The authors conclude that future work should focus on expanding weather diversity, investigating intermediate fusion strategies (e.g., point-cloud fusion), and extending evaluation to closed-loop planning and control.