Case Study: Performance Analysis of a Virtualized XRootD Frontend in Large-Scale WAN Transfers

This paper presents a case study demonstrating that a virtualized XRootD frontend architecture, utilizing a heterogeneous VM cluster with BBR congestion control and TCP extensions, successfully sustained an aggregate throughput of 51.3 Gb/s and peaked at 41.5 Gb/s to Fermilab under high-intensity WAN transfer conditions.

The Gist

Imagine a massive, high-speed library called SPRACE in Brazil. This library doesn't just hold books; it holds the entire digital history of particle physics experiments (like the ones at the Large Hadron Collider). Scientists all over the world need to borrow these "books" (data) instantly.

The problem? The library has a huge back room full of shelves (the Backend), but the front door to the outside world is a single, incredibly wide highway (the WAN). The challenge was figuring out how to get data from the back shelves, through the front door, and onto that highway as fast as possible without causing a traffic jam.

Here is the story of how they did it, explained simply.

1. The Setup: A Team of Delivery Trucks

Instead of using one giant, slow truck to move everything, the library hired a fleet of 8 specialized delivery trucks (these are the XRootD Virtual Machines).

• The Back Room: The data lives on 12 different storage units (like 12 different warehouses). They are connected by a super-fast internal conveyor belt (called pNFS) so the trucks can grab data from anywhere instantly.
• The Trucks: These 8 trucks are virtual (they exist inside a computer cloud), but they are equipped with powerful engines. Some have 10-lane highways, and some have 40-lane highways attached directly to them (using a tech called SR-IOV).
• The Highway: All these trucks merge their cargo onto a massive 100-lane super-highway leading to the rest of the world.

2. The Secret Sauce: Tuning the Engine

In the past, these trucks were driving with "factory settings." Imagine driving a Ferrari with the speed limiter set to 30 mph. The default settings of computer operating systems are usually too cautious for a 100-lane highway. They get confused by the distance and slow down.

The team in this study decided to tune the engine to handle extreme speed:

• The Traffic Cop (BBR): They installed a new, smarter traffic control system called BBR. Instead of just slowing down when it sees a red light, this system constantly measures the road's capacity and pushes the trucks to the absolute limit of what the road can handle without crashing.
• The Cargo Hold (Memory Buffers): They expanded the cargo hold of every truck. Normally, a truck might only carry a few boxes at a time. They increased this limit to hold massive amounts of data, ensuring the trucks never had to stop and wait for the next load.
• The Communication: They taught the trucks to talk to each other faster, using "window scaling" (a way of saying, "I have room for more, send it all!") so data flows like water from a fire hose rather than a dripping tap.

3. The Big Test: The Rush Hour

On a busy morning in October 2025, the library was flooded with requests.

• The Result: The fleet of 8 trucks managed to push data out at a combined speed of 51.3 Gigabits per second. To visualize this: they moved the equivalent of 13 million high-definition movies in just one hour.
• The Star Performer: One specific truck delivering data to a lab in Fermilab (USA) was so efficient that it alone carried 41.5 Gigabits per second. That's like filling a swimming pool with data in seconds.

4. Where Was the Bottleneck?

The researchers wanted to know: What stopped them from going even faster? Was the back room too slow?

• The Back Room: They tested the 12 storage warehouses. Together, they could theoretically push data out at 77 Gigabits per second. So, the shelves were fast enough!
• The Real Limit: The limit wasn't the shelves; it was the trucks. Even with the best tuning, the 8 virtual trucks eventually hit a wall where their own computer brains (CPU) or memory (RAM) got tired from managing so many connections at once. They were running at full capacity, but the back room was still ready to go faster.

5. Did Anything Break?

When you drive that fast, things usually break.

• The Good News: The main route to Fermilab was perfect. Zero failures. The data arrived safely, even at record-breaking speeds.
• The Bad News: About 22% of the total transfers failed, but this wasn't because of the SPRACE library. It was because the destination libraries (where the data was going) were having their own problems. It's like the delivery truck arriving at a warehouse that was locked up for renovations.

The Bottom Line

This paper proves that you don't need a brand-new, super-expensive physical building to move massive amounts of data. By taking a standard virtual setup and tuning the software (like upgrading the traffic lights and expanding the cargo holds), you can squeeze out incredible performance.

They turned a standard computer cluster into a data super-highway, proving that with the right "tuning," virtual machines can handle the heavy lifting of global science.

Technical Summary

Here is a detailed technical summary of the case study "Performance Analysis of a Virtualized XRootD Frontend in Large-Scale WAN Transfers."

1. Problem Statement

High Energy Physics (HEP) experiments, such as the CMS experiment at the LHC, require the efficient transfer of petabytes of data across the Worldwide Computing Grid (WLCG). A primary challenge for Tier-2 computing centers is aggregating data from heterogeneous storage backends to saturate high-capacity Wide Area Network (WAN) links (e.g., 100 Gb/s).

• The Specific Challenge: Standard operating system configurations (defaulting to the CUBIC congestion control algorithm and limited TCP buffers) are insufficient for saturating high-latency, high-bandwidth WAN links.
• The Goal: To document and analyze the performance of the T2_BR_SPRACE storage endpoint, a virtualized architecture designed to aggregate data from a dCache backend and push it to the WAN, specifically under peak production loads.

2. Methodology and Architecture

The study analyzes a decoupled storage architecture deployed at the São Paulo State University (UNESP) facility. The system consists of three main layers:

A. System Architecture

• Backend: 12 dCache storage pools serving data via the pNFS (parallel NFS 4.1) protocol to eliminate single-gateway bottlenecks.
• Frontend: A heterogeneous cluster of 8 XRootD Virtual Machines (VMs).
  • Connectivity: VMs utilize 10 Gb/s and 40 Gb/s network interfaces.
  • Virtualization: 40 Gb/s hosts utilize SR-IOV (Single Root I/O Virtualization) for direct PCI passthrough to network hardware, reducing virtualization overhead.
• Orchestration: Traffic is managed by a local XRootD redirector and orchestrated by the CERN File Transfer Service (FTS).

B. Configuration and Tuning (Key Methodology)

The study emphasizes that performance relies heavily on specific kernel-level optimizations, as default settings fail under high load. Key configurations included:

• Congestion Control: Enabled BBR (Bottleneck Bandwidth and Round-trip propagation time) by Google, replacing the default CUBIC algorithm.
• TCP Buffer Tuning:
  • Increased net.core.rmem_max and wmem_max to a ceiling of 2048 MiB (2 GB).
  • Enabled Window Scaling and Timestamps (RFC 1323) to exceed the 64 KB window limit.
  • Set tcp_adv_win_scale = 1, reserving 50% of the buffer space for the application, effectively advertising a 1024 MiB window to senders while maintaining a safety margin.
• Connection Limits: Increased somaxconn and netdev_max_backlog to 65,536 to handle massive connection queues.

C. Measurement Approach

• Backend Testing: Individual disk I/O tests were conducted on the 12 dCache pools to establish a theoretical maximum throughput.
• Production Monitoring: Data was collected during a high-demand period (October 17, 2025).
• Validation: Local throughput measurements were cross-validated against independent monitoring tools provided by CERN (FTS).

3. Key Results

A. Backend Capacity

• The 12 dCache pools achieved a theoretical aggregate read throughput of 9,670 MB/s (~77.36 Gb/s).
• This confirmed that the disk I/O subsystem was not the bottleneck, as it exceeded the observed WAN output.

B. Aggregate WAN Performance

• Peak Aggregate Throughput: The system sustained a total outbound throughput of 51.3 Gb/s across all 8 VMs.
• Transfer Volume: During the monitored window, 5,696 transfers were initiated, with a 78% success rate.

C. Single-Flow Performance (SPRACE → Fermilab)

• A specific data flow to Fermilab (FNAL) demonstrated exceptional scaling.
• Peak Throughput: Reached 41.48 Gb/s (rounded to 41.5 Gb/s).
• Scaling: The FTS successfully scaled concurrent streams from 300 to 530 during the peak.
• Reliability: This specific high-load flow recorded zero failures, validating the stability of the BBR tuning under extreme stress.
• External Validation: CERN monitoring tools independently confirmed throughput peaks exceeding 40 Gb/s.

4. Discussion and Bottleneck Analysis

• The Bottleneck: The limiting factor was identified as the frontend capacity (the 8 XRootD VMs), not the backend storage.
  • The 51.3 Gb/s aggregate limit suggests saturation of the VM resources (RAM for TCP buffers, CPU contention, or NIC limits) when managing 530 concurrent BBR streams.
  • The backend (77 Gb/s capacity) had significant headroom.
• Reliability Factors: The overall 78% success rate was skewed by a specific destination (cmstnvm1.fnal.gov) which had a 100% failure rate due to issues on the receiving end. Excluding this, major destinations like cmsdcadisk.fnal.gov and eoscms.cern.ch showed >97% success rates.

5. Significance and Contributions

• Validation of Virtualized High-Performance Storage: The study proves that a virtualized XRootD frontend, when properly tuned, can effectively aggregate high-speed storage backends and saturate 100 Gb/s WAN links.
• Critical Role of Kernel Tuning: It provides empirical evidence that default OS network settings are inadequate for HEP data transfers. The combination of BBR congestion control and massive TCP buffer allocation (2 GB ceilings) is essential for achieving >40 Gb/s single-flow performance.
• Operational Blueprint: The paper serves as a reference configuration for other Tier-2 centers aiming to maximize WAN utilization, detailing specific sysctl parameters and architectural setups (SR-IOV, pNFS) required to achieve these results.
• Real-World Stress Test: Unlike theoretical simulations, this study documents performance under real production loads, validating that the architecture can handle the extreme concurrency required by modern particle physics experiments.