LiveStack: OS Support for Cluster-Scale Full-Stack Live Simulation

The paper presents LiveStack, an OS-level approach built on Linux virtualization that enables cluster-scale full-stack simulation with both high fidelity and performance by integrating simulation-oriented scheduling, memory management, IPC, and orchestration to coordinate live and modeled components under shared simulated time.

The Gist

Imagine you are an architect trying to design a massive new city. Before you pour a single drop of concrete or lay a single brick, you want to know exactly how traffic will flow, how the power grid will hold up, and how the whole city will behave when thousands of people move in.

In the real world, building a full-scale test city just to test it is impossible—it's too expensive and takes too long. In the world of computer science, this "city" is a cluster of servers running complex software (like cloud platforms or big data tools).

This paper introduces LiveStack, a new way to simulate these massive computer systems. Here is how it works, explained simply:

The Problem: The "Too Slow" vs. "Too Fake" Dilemma

Previously, scientists had two bad options for testing these systems:

1. The "Slow Motion" Simulator: They could build a perfect, detailed digital model of everything. But it was so slow that simulating just one minute of activity might take days. It was like watching a movie in extreme slow motion; accurate, but useless for quick testing.
2. The "Live" but "Broken" Simulator: They could run the actual software on real computers to make it fast. But they couldn't control when things happened or how different parts talked to each other. It was like letting a real car drive on a test track, but you couldn't stop it or change the traffic lights.

LiveStack solves this by doing both at once: it runs the real software at real speed, but puts it under a "magic clock" that controls the timing perfectly.

The Solution: LiveStack

Think of LiveStack as a super-smart traffic controller built into the operating system (the brain of the computer). Instead of just letting computers run wild, it manages them like a conductor managing an orchestra.

Here are the four main tools LiveStack uses:

1. The "Shared Clock" (Simulation-Oriented Scheduling)

Imagine a group of friends trying to walk together. If one friend walks fast and another walks slow, they get separated.

• How LiveStack works: It gives every part of the simulation a "virtual clock." Even if one computer is running a heavy task and another is idle, the system pauses the fast ones and speeds up the slow ones just enough so they stay in sync. It ensures that if Computer A sends a message to Computer B, Computer B receives it at the exact right moment in the simulation, not just when the real computer happens to be free.

2. The "Private Rooms" (Live Memory Hierarchy Management)

Imagine a busy kitchen where multiple chefs are cooking. If they all use the same cutting board or the same oven, they get in each other's way, and the food takes longer to cook.

• How LiveStack works: It creates "private rooms" (called cells) for each simulated computer. It makes sure that when one simulated computer is running, it doesn't accidentally steal the "cutting board" (memory or cache) from another. If it does, it adjusts the clock to account for the delay, so the simulation stays accurate.

3. The "Controlled Mailroom" (Simulation-Aware IPC)

In a normal computer, messages fly around instantly based on real time. In a simulation, a message might need to wait until a specific time in the story.

• How LiveStack works: It acts like a mailroom that holds letters until the recipient is ready to receive them in the story's timeline. It ensures that a message sent in "minute 5" of the simulation doesn't arrive at "minute 3" just because the real computer was fast. It keeps the story logical.

4. The "Conductor" (Distributed Simulation Orchestration)

When you have many computers working together (a cluster), they need to talk to each other.

• How LiveStack works: It acts as a conductor for a whole orchestra of computers. It coordinates the "private rooms" and the "mailrooms" across different physical machines so that the entire cluster acts like one giant, synchronized system.

The Results: Fast and Accurate

The researchers built a prototype of LiveStack and tested it.

• Speed: They ran a complex database test (TPC-C). A traditional, slow simulator took over a week to finish (and didn't even finish!). LiveStack finished the same test in 90 seconds.
• Accuracy: When they compared LiveStack's results to running the test on real physical hardware, the results were very close (often over 90% accurate).

The Big Picture

LiveStack is a new way to test computer systems. It lets engineers run their real, unmodified software on real hardware, but controls the timing so they can test thousands of different setups quickly.

The paper suggests this is a step toward a future where the computer's operating system itself is built to handle simulations natively, making it much easier to design and test the complex digital systems of tomorrow without needing to build expensive physical test labs.

Technical Summary

Technical Summary: LiveStack: OS Support for Cluster-Scale Full-Stack Live Simulation

1. Problem Statement

Cluster-scale full-stack simulation is critical for evaluating distributed software stacks and emerging hardware components prior to deployment. However, existing methods fail to simultaneously satisfy two essential requirements:

1. Full-Stack Fidelity: The ability to run the unmodified production software stack (OS, runtime, application, coordination layers) without source modification, preserving cross-layer behaviors.
2. Simulation Performance: The ability to complete simulations in a timely manner to support iterative configuration exploration.

Current approaches face specific limitations:

• Discrete-Event Simulators (DES): Tools like gem5 or ns-3 offer high fidelity within specific layers (e.g., microarchitecture or network protocols) but lack full-stack coverage or require replacing the OS/application layers with traffic generators. They also suffer from poor performance due to fine-grained event processing.
• Modular Composition: Frameworks that stitch together different simulators recover full-stack fidelity but are bottlenecked by the slowest component, often failing to complete complex workloads in reasonable timeframes.
• Existing Live Simulation: Systems like Phantora or NEX achieve high performance by running parts of the system on real hardware under simulated time. However, they are often workload-specific, lack full software-stack compatibility (e.g., running without an OS), or do not scale to cluster levels.

The core challenge is that the Linux virtualization stack (KVM/QEMU/libvirt) provides the necessary foundation for near-native execution of unmodified stacks but lacks the OS-level mechanisms to control execution under shared simulated time, isolate co-located components, manage cross-component communication, and orchestrate distributed hosts.

2. Methodology

The authors propose LiveStack, an OS-level approach that transforms the Linux virtualization stack into a live-execution foundation for cluster-scale simulation. The core philosophy is to "keep execution live, and make the OS simulation-aware." LiveStack introduces four subsystems to address the gaps in standard virtualization:

A. Simulation-Oriented Scheduling

This subsystem coordinates live (KVM/QEMU) and modeled components under a shared virtual time with bounded skew.

• vtask Abstraction: Unifies user-space threads (both live vCPUs and modeled components) into a single scheduling unit.
• Virtual-Time Accounting:
  • Live vtasks: Derive virtual time (vtime) from hardware execution by adapting KVM's paravirtual clock (pvclock) to absorb preemption gaps, ensuring vtime advances only during actual vCPU execution.
  • Modeled vtasks: Advance vtime based on performance models reported via ioctl or shared memory.
• Dispatch Logic: The scheduler enforces a skew bound. A vtask executes only if its vtime is within the configured skew of the scope's minimum vtime. Blocked vtasks are excluded from the minimum calculation to prevent deadlocks, and their time is forwarded upon resumption to maintain causality.

B. Live Memory Hierarchy Management

This subsystem addresses performance interference among co-located live components on shared hardware (C2).

• Cell Abstraction: A simulation-oriented generalization of Linux cpusets that binds a live host to a controlled resource domain.
• Spatial Isolation: Assigns concurrent live hosts to distinct cells, controlling CPU/NUMA placement, cache capacity (via Intel CAT/AMD QoS), and memory bandwidth (via MBA).
• Temporal Reconditioning: At cell switch boundaries, the system invalidates the outgoing cell's cache footprint and prefetches the incoming cell's working set. Residual deviations are estimated via Performance Monitoring Unit (PMU) sampling and explicitly incorporated into the virtual-time advance, rather than hidden.

C. Simulation-Aware IPC

This subsystem ensures cross-component messages (between live and modeled parts) are delivered in simulated-time order, not host wall-clock order (C3).

• Architecture: Organized around Messages (separating timing metadata from payload), Endpoints (proxies for interfaces like QEMU devices), and Hubs (kernel-resident routers).
• Mechanism: Messages are queued in incoming queues ordered by visibility time. The kernel hub handles lightweight routing and latency control, with extensibility via eBPF hooks. This prevents causality violations where a receiver might see an event before the sender's simulated time has reached that point.

D. Distributed Simulation Orchestration

This subsystem scales the per-host mechanisms across a physical cluster (C4).

• Hybrid Control: Combines a per-host user-space daemon (for control-plane tasks like placement and channel setup) with kernel-level support (for hot-path tasks like vtime updates and IPC).
• Semantics Preservation: Remote vtasks are represented as local proxy vtasks to enforce bounded-skew rules across machines. Logical hubs are implemented as distributed instances exchanging metadata.
• Optimization: Reduces cross-host traffic by co-locating frequently interacting components when resources permit.

3. Key Contributions

1. OS-Level Live Simulation: LiveStack is the first system to integrate simulation control and orchestration directly into the OS kernel (specifically the Linux virtualization stack), moving beyond user-space composition layers.
2. Unified Subsystems: It defines a substrate comprising scheduling, memory management, IPC, and orchestration that allows live and modeled components to coexist under shared simulated time while controlling interference.
3. Full-Stack Fidelity with Performance: By leveraging KVM/QEMU for native execution and adding OS-level controls, it achieves the ability to run unmodified production stacks at near-native speeds, a combination previously unattainable.

4. Preliminary Results

The authors implemented a prototype and evaluated it against a physical testbed and a modular gem5-based setup.

• Workloads: Evaluated on CoreMark, TPC-C (MySQL), YCSB (HBase), and TPC-DS (Spark/Hive).
• Performance:
  • LiveStack completed the TPC-C (MySQL) workload in 90.4 seconds.
  • A comparable modular gem5 setup failed to finish the same workload within a week.
  • LiveStack showed slowdowns ranging from 1.1× (CoreMark) to 4.0× (TPC-DS on Spark) relative to physical execution, depending on the number of synchronized instances.
• Accuracy: The prototype reproduced physical behavior with high accuracy (e.g., 97.8% arithmetic speed for CoreMark, 92.3% throughput for TPC-C).
• Limitations: The authors note that inaccuracies remain due to uncontrolled memory-hierarchy interference and IPC visibility timing in the current prototype, as not all design subsystems are fully implemented.

5. Significance and Claims

The paper claims that LiveStack demonstrates the feasibility of cluster-scale full-stack simulation by combining the high performance of live execution with the fidelity of modular composition.

• Shift in OS Responsibility: The authors position LiveStack as a step toward "simulation-native OS support," where simulation control and orchestration become core OS abstractions rather than user-space services running atop a simulation-agnostic kernel.
• Practical Utility: It enables operators to evaluate deployment configurations and designers to assess emerging hardware (accelerators, memory systems) before they are available at scale, without the cost and impracticality of building physical testbeds.
• Modesty: The authors explicitly state that the current prototype is preliminary. They acknowledge that several subsystems are under active development and that the system currently implements only part of the full design, with future work needed to address remaining sources of inaccuracy and integrate other virtualization technologies.