💻 computer science

CARLAverse: A Highly Modular, Distributed, and Multimodal Framework for Human-in-the-Loop Simulation

CARLAverse is an open-source, distributed, and multimodal simulation framework that integrates driving, cycling, and pedestrian environments to enable scalable human-in-the-loop experiments by decoupling latency-critical haptic feedback from network bottlenecks through a novel local-client/central-server architecture.

Original authors: Patrick Rebling, Philipp Nenninger, Reiner Kriesten

Published 2026-09-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Patrick Rebling, Philipp Nenninger, Reiner Kriesten

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

To build cars that can drive themselves safely, engineers must teach them how to navigate a world filled with unpredictable people. While computers are excellent at following strict rules, they often struggle with the subtle, unspoken negotiations that happen on a busy street: a cyclist glancing over their shoulder before turning, a pedestrian hesitating at a crosswalk, or a driver signaling a lane change. To study these complex interactions without risking real lives, researchers use simulators. These digital environments allow humans to act out driving scenarios while computers record their reactions. However, a major hurdle remains: most current simulators are isolated islands. They can show a human driver a virtual world, but they struggle to connect that driver with other real humans acting as cyclists or pedestrians in the same shared space. When multiple people try to interact in a virtual world over a network, the delay in sending information can break the illusion, making the experience feel sluggish or causing physical discomfort.

A team of researchers at Karlsruhe University of Applied Sciences has addressed this challenge by creating a new, open system called CARLAverse. This framework acts as a bridge, allowing different types of simulators—those for cars, bicycles, and pedestrians—to connect and interact in a single, synchronized virtual city. The core innovation lies in how the system handles the physics of movement. Instead of sending every tiny detail of a person's steering or pedaling to a central computer and waiting for a reply, the system splits the work. Each participant's local machine calculates the immediate, high-speed forces they feel, such as the resistance in a steering wheel or the vibration of a bicycle handlebar, while a central server manages the broader traffic flow and the positions of other vehicles. This design ensures that the feedback a human feels is instant and realistic, even if the participants are located in different buildings or cities. By combining these distributed calculations with a shared virtual environment, the researchers have created a platform where multiple people can safely practice and study how they react to one another in mixed traffic, capturing the genuine social dynamics that artificial intelligence often misses.

The system is built on a foundation of modularity, meaning it can adapt to various hardware setups without requiring a complete rebuild. It supports three distinct modes of travel: driving a car, riding a bicycle, and walking. For the car, the system connects to standard driving simulators with steering wheels and pedals. For the bicycle, it links to smart trainers that can adjust resistance to mimic hills or wind. For pedestrians, it uses virtual reality headsets and cameras to track body movements, allowing a person to walk naturally and see the world from a human height. What makes this approach unique is its ability to keep these different experiences in perfect sync. When a pedestrian in virtual reality steps off a curb, the driver in the car simulator sees that movement instantly and feels the appropriate change in the road environment. The system achieves this by carefully separating the heavy lifting of global traffic management from the sensitive, fast-paced calculations of individual movement.

In the past, connecting multiple simulators often relied on software layers that acted as middlemen, passing messages back and forth. The researchers found that these middlemen introduced too much delay, which is unacceptable for systems that rely on physical feedback like force feedback or motion platforms. To solve this, CARLAverse bypasses these extra layers, allowing the local machines to talk directly to the central server using streamlined communication channels. This direct line of communication ensures that when a driver turns the wheel, the virtual car responds immediately, and the force feedback motor pushes back against their hand without a noticeable lag. Similarly, when a cyclist pedals, the resistance adjusts in real-time to match the virtual terrain. This precision is vital because even a tiny delay can cause a person to feel sick or lose their sense of presence in the simulation.

The researchers also integrated advanced features to make the experience more immersive. The system generates sound based on the location of the user, so a driver hears the engine roar differently depending on where they are in the city, and a pedestrian hears an approaching car from the correct direction. Visuals are handled with care to ensure that the images on screens or in headsets align perfectly with the user's perspective, preventing the dizziness that often plagues virtual reality. For the bicycle simulator, the system even calculates the physical forces of wind and road slope to adjust the resistance on the trainer, making the ride feel authentic. For pedestrians, the system uses standard webcams to track body movements, translating the way a person leans or gestures into the virtual world, allowing for natural interactions like waving or looking around.

The ultimate goal of this work is to provide a tool for understanding human behavior in traffic. By placing real people in a shared virtual space, researchers can observe how they negotiate right-of-way, how they signal their intentions, and how they react to unexpected events. This data is difficult to gather with artificial agents alone, as computers often fail to replicate the subtle nuances of human body language and hesitation. The CARLAverse framework allows for large-scale experiments where multiple institutions can participate simultaneously, connecting simulators across wide distances. This capability opens the door to studying complex traffic scenarios involving cars, bikes, and people all at once, providing a richer understanding of the social fabric of our roads. While the system is currently a simulation, the researchers plan to validate its physical models further and expand its capabilities, including adding motion platforms for bicycles and omnidirectional treadmills for pedestrians. By making this technology open and accessible, the team hopes to lower the barriers for other researchers to conduct similar studies, fostering a collaborative effort to make future transportation systems safer and more attuned to human needs.

Technical Summary: CARLAverse

Problem Statement

The development of autonomous driving systems requires rigorous testing in mixed-traffic scenarios involving Vulnerable Road Users (VRUs) such as cyclists and pedestrians. While purely artificial agents (NPCs) can simulate traffic flow, they often fail to capture authentic human social negotiations, subtle behaviors like eye contact and yielding, and non-verbal cues. Consequently, Human-in-the-Loop (HITL) simulation is essential for studying these interactions safely. However, existing multi-agent HITL platforms face significant technical barriers:

Latency and Synchronization: High-fidelity haptic feedback (force feedback, motion cueing) requires update rates often exceeding 100 Hz. Centralized architectures that route these high-frequency control signals through middleware (e.g., ROS/DDS) introduce serialization overhead and network jitter, degrading physical immersion and potentially causing simulator sickness.
Hardware Integration: Commercial connected simulators are often expensive and proprietary, while open-source alternatives frequently lack the native support for physical immersion or the streamlined hardware abstraction needed for cohesive multi-actor ecosystems.
Scalability: Existing frameworks often struggle to support geographically distributed, heterogeneous nodes (e.g., combining a vehicle cockpit, a bicycle trainer, and a pedestrian VR setup) without compromising the synchronization required for valid behavioral data.

Methodology

The paper introduces CARLAverse, an open-source, multimodal simulation ecosystem built upon the CARLA driving simulator. The core methodology relies on a distributed physics architecture and a modular hardware abstraction framework designed to decouple latency-critical local computations from global network synchronization.

1. Distributed Physics Architecture

Unlike traditional centralized simulators where a single server computes all physics, CARLAverse structurally decouples the simulation:

Central Server: A dedicated CARLA server acts as the authoritative orchestrator. It computes global traffic management, NPC physics, weather conditions, and the global collision matrix. It synchronizes the world state but does not compute the local dynamics of human participants.
Local Nodes: Each human participant operates on a dedicated local node (e.g., a specific PC for a driver, cyclist, or pedestrian). These nodes compute ego-physics locally. This includes tire-road interactions, self-aligning torques for force feedback, motion cueing algorithms, and pose estimation.
Communication: The system bypasses heavy middleware (like ROS) for active control. Instead, it uses direct CARLA API RPCs over TCP for state updates and connectionless UDP for high-frequency telemetry (force feedback, motion commands). This eliminates the "publish-subscribe" overhead, reducing the haptic loop latency to strictly local components ( $T_{local} = t_{input} + t_{physics,local} + t_{actuation}$ ).

2. Modular Hardware Abstraction

The ecosystem utilizes a Hardware Abstraction Layer (HAL) to ensure hardware agnosticism:

Configuration-Driven: Simulator instantiation is governed by YAML configuration files, allowing rapid reconfiguration from desktop setups to high-fidelity motion platforms without source code modification.
Specialized Subsystems:
- Force Feedback (FFB): A standalone C++ server translates physics calculations into DirectInput/Evdev commands via a tightly packed 40-byte UDP payload. It includes a hardware watchdog to ensure safety.
- Motion Cueing: A separate C++ server translates vehicle telemetry (roll, pitch, heave) into motion platform commands (e.g., D-BOX), utilizing washout filters to maintain plausibility.
- Spatial Audio: A centralized engine (CARLAcoustics) generates context-aware, multi-channel soundscapes based on the user's position, with rendering performed locally.

3. Multimodal Simulation Nodes

The framework integrates three distinct interaction domains:

DrivoCARLA (Vehicle): Supports scalable immersion from desktop to full cabin mockups. It features a custom control pipeline with non-linear mapping for steering/pedals, a state machine for vehicle systems (lights, indicators), and integration with ADAS testing.
CycloCARLA (Bicycle): Overrides CARLA's native four-wheeled physics with a custom single-track kinematic bicycle model. It captures high-frequency wheel speed via a microcontroller (bypassing standard BLE latency) and uses a 2D Lookup Table to translate physical resistance forces into smart trainer commands.
WalkoCARLA (Pedestrian): A VR-based node using OpenXR for hardware abstraction. It employs a vision-based pose estimation pipeline (MediaPipe) to extract 33 skeletal landmarks from a standard webcam, transforming them into the CARLA avatar's coordinate system to capture authentic body language and intent.

Key Contributions

Distributed Physics Paradigm: The primary methodological contribution is the decoupling of ego-physics from the central server. By localizing latency-critical calculations (haptics, motion, pose), the system supports high-frequency control loops necessary for realistic immersion while tolerating Wide Area Network (WAN) latency for global synchronization.
Open-Source Multimodal Ecosystem: CARLAverse provides an open-source framework that integrates vehicle, bicycle, and pedestrian simulators into a shared virtual environment. While other frameworks like "Simulation for All" (2025) have integrated multiple agent types, CARLAverse distinguishes itself through its distributed physics architecture and hardware-agnostic design, enabling the study of genuine human-human interactions in mixed traffic without the rigid hardware dependencies of prior solutions.
Hardware-Agnostic Abstraction: Through the use of YAML configuration, custom C++ servers for FFB and motion, and OpenXR/SDL interfaces, the system allows researchers to integrate diverse hardware (motion platforms, smart trainers, VR headsets) without proprietary lock-in.
Direct API Orchestration: The architecture deliberately avoids routing high-frequency control signals through middleware like ROS, instead utilizing direct RPC and UDP streams to minimize jitter and serialization latency.

Results and Validation

The paper primarily presents the architecture, implementation, and integration principles of the CARLAverse.

System Deployment: The framework has been successfully deployed to support geographically distributed, cross-institutional co-simulation, allowing researchers at different institutions to interact simultaneously in a shared virtual intersection.
Performance Claims: The distributed architecture is designed to eliminate network transmission delays from the time-critical haptic loop. The use of direct UDP streams and local physics calculation is claimed to support the high update rates (hundreds of Hz) required for stable haptic feedback and motion cueing.
Validation Status: The paper explicitly states that detailed quantitative validation of individual physical and perceptual subsystems (e.g., the mathematical accuracy of the bicycle dynamics, the acoustic modeling of CARLAcoustics, and the specific latency measurements) are addressed in dedicated companion publications and ongoing work. A preliminary user study for the DrivoCARLA pipeline is referenced as having demonstrated empirical utility.

Significance and Claims

The paper positions CARLAverse as a critical step toward ecologically valid mixed-traffic research. Its significance lies in:

Bridging the Behavioral Gap: By enabling multiple human participants to interact in a shared space, it captures social negotiations and non-verbal cues that AI-driven NPCs currently fail to replicate.
Scalability and Accessibility: The open-source, hardware-agnostic nature lowers financial and technical barriers, facilitating cross-institutional collaboration and reproducible experiments that were previously limited to expensive, proprietary commercial setups.
Methodological Rigor: The distributed physics approach offers a solution to the fundamental trade-off between localized haptic realism and centralized environmental synchronization, enabling scalable, connected multi-agent experiments without compromising physical immersion.

The authors conclude that while the foundational architecture is operational, future work will focus on empirical validation of the physical models, hardware extensions (e.g., omnidirectional treadmills), and large-scale HITL experiments using standardized OpenSCENARIO files to investigate yielding behaviors and trust dynamics.