VLCs: Managing Parallelism with Virtualized Libraries

This paper introduces Virtual Library Contexts (VLCs), a process-level mechanism that encapsulates software libraries and their resource allocations to enable safe, high-performance parallel execution of non-composable or thread-unsafe libraries without requiring modifications to the library code or operating system.

The Gist

Imagine you are running a busy kitchen (your computer program). In this kitchen, you have several different specialized chefs (software libraries) working at the same time. One chef is an expert at chopping vegetables (linear algebra), another is great at baking (machine learning), and a third is a master of grilling (parallel processing).

The Problem: The "Me-Too" Kitchen
In a standard kitchen, every chef assumes they own the entire room. If the vegetable chef needs 4 hands to chop, they grab 4 hands. If the baker needs 4 hands to knead dough, they grab 4 hands. If you have 8 hands total (8 CPU cores) and both chefs try to use all 8 at once, they end up bumping into each other, fighting for space, and slowing everything down. This is called resource contention.

Usually, to fix this, you'd have to:

1. Fire the chefs and hire new ones who know how to share (rewrite the library code).
2. Build a new kitchen with separate rooms for each chef (run separate computer programs).
3. Hire a manager to force them to share (modify the operating system).

All of these are hard, expensive, or impossible if you didn't build the kitchen yourself (using legacy or closed-source software).

The Solution: Virtual Library Contexts (VLCs)
The paper introduces VLCs, which act like invisible, magical partitions inside your single kitchen.

Think of a VLC as a personalized "force field" around a specific chef.

• The Partition: You can tell the vegetable chef, "You get the left side of the counter (cores 0–3)." You tell the baker, "You get the right side (cores 4–7)."
• The Illusion: Inside their force field, the vegetable chef thinks they are the only one in the kitchen with access to the whole counter. They don't know the baker exists. This stops them from fighting over resources.
• No Renovation Needed: You don't have to change the chefs' recipes (library code) or rebuild the kitchen (OS). You just put up the invisible walls.

Why This is Cool (The Magic Tricks)

1. The "Double Trouble" Trick (Thread Safety):
   Some chefs are notoriously grumpy and will get confused if two people try to talk to them at once (thread-unsafe code). Usually, you can only have one of these chefs working at a time.
   With VLCs, you can load two copies of the same grumpy chef into the kitchen. Because they are in different force fields (VLCs), they don't see each other. They both think they are the only one there, so they can work in parallel without crashing the kitchen.

2. The "Smart Manager" (Auto-Tuner):
   Figuring out exactly how many hands each chef needs is tricky. If you give the baker too many hands, they waste energy. Too few, and the dough doesn't get kneaded fast enough.
   The paper's VLCs come with a smart assistant that automatically tries different combinations of hand allocations to find the perfect balance, so you don't have to guess.

What the Paper Actually Found
The researchers built prototypes of this system for C++ and Python and tested it with real-world tools like OpenMP, OpenBLAS, and LibTorch. Here are their specific claims:

• Speed: In tests, using VLCs made programs run up to 2.85 times faster than the standard way of doing things.
• Specific Wins:
  • A machine learning task (hyperparameter tuning) got a 6.43x speedup when VLCs prevented the different tasks from stepping on each other's toes.
  • A 3D heat simulation using multiple graphics cards (GPUs) ran 1.46 times faster with VLCs compared to the old method of splitting the work across multiple computer programs.
  • Solving complex math problems (eigenvalues) with a library called ARPACK (which usually can't run two tasks at once) saw a 1.96x speedup by running two isolated copies in parallel.
• Low Cost: The "magic walls" (VLCs) add almost no extra work to the computer. The overhead was less than 0.54% in most cases.
• Ease of Use: You don't need to be a coding wizard. The researchers showed that you can add this power to existing apps by changing as few as 10 to 27 lines of code.

In Summary
VLCs are a tool that lets you slice a single computer program into smaller, isolated "virtual rooms." This lets different software libraries work together peacefully without fighting for resources, without needing to rewrite their code, and without the heavy cost of running separate programs. It's like giving every chef in the kitchen their own private workspace while still letting them share the same ingredients.

Technical Summary

Technical Summary: VLCs: Managing Parallelism with Virtualized Libraries

Problem Statement

Modern parallel applications increasingly rely on the composition of external software libraries to encapsulate and exploit parallelism. However, many widely used libraries (e.g., OpenMP, OpenBLAS, LibTorch, TBB) are not designed with composition in mind. They often assume exclusive access to all system resources or allow configuration only at the process level (e.g., via environment variables). When multiple such libraries are composed within a single process, they frequently over-allocate resources, leading to severe thread oversubscription, resource contention, and degraded performance.

Existing solutions to manage cross-library interference typically require modifying the library source code, the operating system kernel, or adopting specific runtimes (e.g., Lithe, Bolt). These approaches are often infeasible for legacy scientific software stacks, closed-source libraries, or managed computing clusters where users lack root privileges. Furthermore, current alternatives often force the use of multiple processes with explicit inter-process communication (IPC), sacrificing the efficiency of shared memory and data sharing inherent to a single-process architecture.

Methodology: Virtual Library Contexts (VLCs)

The authors propose Virtual Library Contexts (VLCs), a user-space, lightweight library virtualization layer that encapsulates sets of libraries and their associated resource allocations within a single process. VLCs allow users to partition resources (e.g., CPU cores, GPUs) between libraries without modifying library code or recompiling binaries.

Core Mechanisms

1. Library Isolation via Linker Namespaces:
   VLCs utilize linker namespaces (created via dlmopen in C++) to isolate libraries. This prevents symbol conflicts when multiple copies of the same library (or incompatible versions) are loaded into a single process. Each VLC maintains its own static state, enabling parallel execution of libraries that are otherwise not thread-safe.

2. Resource Virtualization:
   The VLC runtime intercepts system calls and file system accesses used by libraries to query system resources (e.g., sched_getaffinity, reading /proc/cpuinfo). A VLC Monitor process, operating entirely in user space via ptrace, intercepts these calls. It modifies the responses to reflect only the resources assigned to the specific VLC.

   • CPU Affinity: The monitor sets CPU affinity for all threads spawned within a VLC to ensure they run exclusively on allocated cores.
   • File Interposition: Resource configuration files are redirected to virtual versions listing only the assigned resources.
   • Optimization: Seccomp BPF is used to filter non-relevant system calls, minimizing the overhead of interception.

3. Handling Incompatible Libraries (Service VLC):
   Some libraries (e.g., TBB, CUDA, older pthreads) are incompatible with dlmopen because they internally call dlopen. To address this, VLCs introduce a Service VLC. Incompatible libraries are loaded globally into this special context. A "shim" layer is generated at compile time for dependent libraries, which redirects API calls to the globally loaded instance in the Service VLC, maintaining isolation while ensuring compatibility.

4. Implementation:
   The authors implemented prototypes in C++ (VLC++) and Python (PyVLC).

   • VLC++: Uses dlmopen, ptrace, and Seccomp BPF. It provides a "transparent mode" using generated shims to redirect function calls without requiring developers to manually manage function pointers (dlsym).
   • PyVLC: Overrides Python's __import__ and getattr mechanisms to route module imports to specific VLC contexts. It manages internal caches (e.g., sys.modules) to ensure state isolation.

5. Auto-Tuning:
   Recognizing that finding optimal resource partitions is non-trivial, VLCs include an auto-tuner that performs a grid search over the configuration space to identify optimal resource allocations, helping users avoid counterintuitive performance pitfalls.

Key Contributions

• Library-Level Virtualization: Introduction of a novel concept that offers fine-grained resource management without requiring modifications to library code or the OS kernel.
• Performance Isolation without Data Isolation: VLCs provide performance isolation (preventing contention) while maintaining the efficiency of a shared address space for data sharing, unlike containers or VMs.
• Enabling Parallelism: VLCs enable the parallel execution of thread-unsafe libraries by loading multiple instances with distinct static states and allow for the optimization of nested parallelism by tailoring resource allocation to specific tasks.
• Prototypes: Implementation of C++ and Python prototypes demonstrating applicability across compiled and interpreted languages.

Experimental Results

The authors evaluated VLCs on a machine with 24 CPU cores and 4 GPUs, using benchmarks involving OpenMP, OpenBLAS, LibTorch, Kokkos, and ARPACK.

• Overhead: The overhead of VLCs is minimal. System call interception adds negligible latency (e.g., ~20µs for sched_getaffinity), and the total application overhead is consistently below 1% (Table 4).
• Contention Reduction: In a hyperparameter tuning workflow using LibTorch, VLCs achieved a 2.85× speedup over the default concurrent configuration and a 6.43× speedup over sequential tuning by preventing CPU oversubscription.
• Nested Parallelism: For a workload mixing large and small matrix multiplications (GEMMs), VLCs achieved a 1.58× speedup over a sequential baseline by dedicating specific core subsets to different task sizes, a configuration impossible with standard process-level settings.
• Thread-Unsafe Libraries: By loading two instances of the thread-unsafe ARPACK library into separate VLCs, the system achieved a 1.96× speedup (C++) by running eigenvalue computations concurrently.
• Multi-GPU Support: For the Heat3D application using Kokkos, VLCs enabled single-process multi-GPU execution on legacy versions of Kokkos (pre-4.3). This approach achieved performance identical to native multi-GPU support in newer versions and a 1.46× speedup over traditional MPI-based multi-process implementations, while requiring significantly less code migration effort.

Significance and Claims

The paper claims that VLCs offer a practical, low-overhead solution to the "composition problem" in parallel computing. By virtualizing resources at the library level, VLCs allow developers to:

1. Prevent Contention: Partition resources to avoid oversubscription without modifying library code.
2. Optimize Nested Parallelism: Dynamically allocate resources to different phases of a computation based on their specific needs.
3. Parallelize Unsafe Code: Safely execute concurrent calls to libraries that are not thread-safe by isolating their static states.
4. Simplify Migration: Provide a path for legacy applications (e.g., MPI-based Kokkos) to utilize modern hardware features (like multi-GPU) without extensive refactoring.

The authors emphasize that VLCs do not require kernel modifications, making them deployable on managed clusters and supercomputers where users lack root access. They position VLCs as a complementary tool that bridges the gap between the rigid resource management of current libraries and the flexible needs of complex, composed scientific applications.