Enhancing OLAP Resilience at LinkedIn

This paper presents a holistic resiliency framework for Apache Pinot at LinkedIn, featuring Query Workload Isolation, Impact-Free Rebalancing, Maintenance Zone Awareness, and Adaptive Server Selection, which collectively ensure stable subsecond query latency and high availability for petabyte-scale OLAP workloads under failures and load spikes.

The Gist

Imagine LinkedIn as a massive, high-speed library where millions of people are constantly asking questions about data (like "Who viewed my profile?" or "What ads should I show?"). This library is so big it holds petabytes of books (data), and it needs to answer questions in the blink of an eye (sub-second latency).

The system running this library is called Apache Pinot. But like any giant library, it faces three big problems:

1. The "Noisy Neighbor" Problem: One person asking a super-hard question can tie up the librarian, making everyone else wait.
2. The "Moving Shelves" Problem: When the library expands or fixes a broken shelf, moving the books usually causes chaos and stops people from finding things.
3. The "Slow Librarian" Problem: If one librarian is having a bad day (slow computer), the whole group of librarians assigned to that section slows down, even if the others are fast.

This paper describes how LinkedIn built three "superpowers" to fix these problems and keep the library running smoothly, even when things go wrong.

---

1. The "Fairness Budget" (Query Workload Isolation)

The Problem: Imagine a shared kitchen where everyone is cooking. If one person decides to deep-fry a whole turkey (a heavy, complex query), they might use up all the stove burners and oil. Suddenly, the person trying to make a quick sandwich (a simple, fast query) has to wait forever. This is the "noisy neighbor" problem.

The Solution: LinkedIn introduced Query Workload Isolation (QWI).

• The Analogy: Think of this as giving every group of cooks a personal budget of stove-time and oil.
• How it works: Before a cook starts, the system checks their budget. If they try to use more oil than they have left, the system gently stops them before they clog up the stove.
• The Magic: It does this so fast (in less than a millisecond) and so efficiently that it doesn't slow down the good cooks. It ensures that if one person tries to hog resources, the rest of the library keeps running at full speed. It's like having a bouncer who stops the rowdy party-goers from blocking the exit, without kicking out the polite guests.

2. The "Smart Moving Day" (Maintenance Zone Awareness & Impact-Free Rebalancing)

The Problem: Libraries need to move books around when they add new shelves or fix broken ones. Usually, this is a disaster: you have to take books off the shelves, move them, and put them back. While you're doing this, the books are missing, and people can't find them. Also, if you move all the books from one building to another, and that building catches fire (a "zone failure"), you lose everything.

The Solution: LinkedIn created a Smart Moving Day strategy.

• The Analogy: Imagine the library is spread across different buildings (Maintenance Zones).
  • Part A (Smart Placement): When you get new books, you don't just throw them in the nearest building. You spread them out so that if one building burns down, you still have copies of every book in the other buildings. The system uses a "greedy swap" algorithm to shuffle books around until they are perfectly balanced across all buildings.
  • Part B (Impact-Free Moving): When it's time to move books to a new shelf, the system doesn't just grab them and run. It first tells the librarians: "Stop asking for these specific books for a second." Once the librarians stop asking, the movers quietly swap the books. Then, the librarians start asking again.
• The Magic: Because the system waits for people to stop asking before moving the heavy stuff, nobody notices the move. The library never goes offline, and no one loses data, even during massive upgrades.

3. The "Smart GPS" (Adaptive Server Selection)

The Problem: Imagine a delivery service with 100 drivers. The dispatcher usually sends packages to drivers in a circle (Round Robin). But what if Driver #5 has a flat tire and is moving at 1 mph? The dispatcher keeps sending packages to Driver #5 because it's their turn. The whole delivery is delayed because of one slow driver.

The Solution: LinkedIn built an Adaptive Server Selection (ADSS) system.

• The Analogy: Instead of a rigid circle, the dispatcher now has a live GPS for every driver.
• How it works: The system constantly checks: "Who is fast right now? Who is stuck in traffic?" If Driver #5 is slow, the system instantly stops sending packages to them and routes them to Driver #6, who is zooming along.
• The Magic: It's not just about avoiding the slow driver; it's about predicting traffic. If Driver #6 looks like they are about to get stuck, the system sends the package to Driver #7 instead. This happens automatically and instantly. Even if a driver has a sudden "flat tire" (a computer glitch), the system reroutes traffic in seconds, so the customers (users) never feel the delay.

---

The Big Picture

By combining these three tools, LinkedIn has built a library that is:

1. Fair: No single user can crash the system for everyone else.
2. Resilient: You can move shelves, fix buildings, or upgrade the roof without ever closing the doors.
3. Smart: It instantly reroutes traffic away from trouble spots, keeping everything running at top speed.

These aren't just theoretical ideas; they are used every day by millions of LinkedIn users, ensuring that when you check your feed or search for a job, the answer comes back instantly, no matter what's happening behind the scenes.

Technical Summary

Here is a detailed technical summary of the paper "Enhancing OLAP Resilience at LinkedIn."

1. Problem Statement

Modern enterprises rely on real-time OLAP (Online Analytical Processing) systems like Apache Pinot to power interactive analytics on petabyte-scale datasets with sub-second latency requirements. As these systems become integral to service architectures, maintaining strict Service Level Agreements (SLAs) during failures, load spikes, and cluster changes is as critical as raw performance.

The paper identifies four primary challenges in operating large-scale OLAP clusters:

1. Noisy-Neighbor Interference: In shared clusters, heterogeneous workloads (e.g., interactive dashboards vs. batch metrics) contend for CPU and memory. A single complex query can saturate resources, causing "tail latency" spikes and SLA violations for co-located workloads.
2. Disruption During Maintenance: Routine operations (upgrades, scaling, recovery) require data redistribution. Without awareness of fault domains (Maintenance Zones), moving data can degrade query performance or leave the system vulnerable if an entire zone is drained.
3. Inefficient Routing: Traditional round-robin routing within replica groups fails when a single server in a group becomes slow (due to GC pauses or I/O contention). Since brokers wait for all servers in a group to respond, one slow node degrades the latency for the entire query.
4. Lack of Holistic Resilience: Existing solutions often address these issues in isolation. The paper argues for a unified framework that combines resource isolation, intelligent placement, and adaptive routing.

2. Methodology and Key Contributions

The authors present four interconnected mechanisms developed for Apache Pinot at LinkedIn to create a holistic resilience framework.

A. Query Workload Isolation (QWI)

• Concept: QWI introduces workload-level CPU and memory budgeting across the Broker and Server tiers. It treats each workload as a first-class entity with enforceable budgets.
• Mechanism:
  • Resource Accounting: Uses a sampling-based approach (Algorithm 1) where worker threads periodically write CPU time and memory metrics to thread-local regions. A dedicated sampler aggregates these metrics with lock-free operations, achieving <0.3% CPU overhead.
  • Enforcement: Budgets are propagated from the Controller to hosts. The system performs admission control (rejecting queries if budget is insufficient) and in-execution enforcement (killing queries that exceed thresholds).
  • Granularity: Supports table-level and tenant-level propagation. It handles "runaway" queries by killing them at 85% heap usage and all queries at 99% to prevent OOM crashes.
• Goal: Prevents resource starvation and ensures fair sharing, delivering predictable tail latency with <1% overhead.

B. Maintenance Zone (MZ) Aware Data Placement

• Concept: Ensures data replicas are distributed across fault domains (Maintenance Zones) to maintain availability even if an entire zone is drained for maintenance.
• Mechanism:
  • Greedy Swap Algorithm (Algorithm 3): When nodes are added or removed (e.g., during scaling), the algorithm detects "bad" states where replicas of the same segment group are concentrated in one MZ. It iteratively swaps instances between "bad" rows and other rows to minimize the number of overpopulated zones while minimizing data movement.
  • Correctness: The paper provides an inductive proof that the algorithm converges to a state where no more than $\lceil R/MZ \rceil$ replicas of a segment are lost if a single MZ fails.
• Goal: Guarantees data availability during planned maintenance without requiring rigid, static zone assignments.

C. Impact-Free Rebalancing

• Concept: A procedure to transition data from a current placement to a target placement (e.g., after MZ-aware assignment) without degrading query SLAs.
• Mechanism:
  • State Model: Tracks Initial, Desired, Added, and Removed segments per host.
  • Two-Phase Execution:
    1. Rebalancing Step: Drains queries from a host, waits for in-flight queries to finish, moves substantial data, and re-enables traffic.
    2. Progress Step: If no host can be safely drained (due to low replication headroom), the system adds a small percentage of segments (similar to a daily data push) across all hosts to restore replication levels.
  • Safety: Ensures that at no point does the number of available replicas for any segment drop below a configurable threshold (e.g., $R-1$).
• Goal: Enables zero-downtime scaling and upgrades, moving terabytes of data without impacting query latency or completeness.

D. Adaptive Server Selection (ADSS)

• Concept: Replaces static round-robin routing with dynamic selection based on real-time load and performance signals.
• Mechanism:
  • Mirror Server Sets (MSS): Instead of routing to a random replica group, the broker selects the best server within a specific Mirror Server Set (servers holding identical data).
  • Scoring Algorithms:
    1. In-flight requests: Counts pending tasks.
    2. Latency EMA: Exponential Moving Average of response times.
    3. Hybrid: Combines estimated queue depth ($Q_{EMA}$) and latency ($Latency_{EMA}$) with a cubic penalty to prevent herd behavior.
  • Oscillation Control: Uses a Softmax probability policy (Eq. 4) to introduce calibrated randomness, preventing all brokers from synchronizing on the same "best" server and causing load swings.
• Goal: Routes queries away from slow or failing nodes within seconds, preserving low-latency guarantees even during partial failures.

3. Results and Evaluation

The mechanisms were deployed in LinkedIn's production environment, spanning ~200 clusters, 10,000+ server pods, and ~10 PB of data.

• QWI Performance:

  • Overhead: Added <1% CPU overhead and ~0.5 GB heap usage. P99 latency remained unchanged.
  • Isolation: In experiments, a compute-heavy workload previously caused co-tenant P95 latency to spike from 500ms to >6s. With QWI, the co-tenant remained stable at ~500ms while the heavy workload was throttled.
  • Accuracy: CPU budget enforcement accuracy was 95–100%; memory accuracy was 90–100%.

• MZ-Aware Placement & Rebalancing:

  • Availability: Sustained >99.9% availability despite 7% daily node churn.
  • Impact: Before MZ-aware placement, draining a zone could leave 33% of data missing. With the new system, zone drains completed with zero impact on query completeness or latency.
  • Migration: Successfully migrated 10 PB of data to cloud-native infrastructure in 6 months with zero SLA violations.

• Adaptive Server Selection (ADSS):

  • Latency: At P98, total query latency improved by ~9% by avoiding slow nodes.
  • Failure Handling: Detected and routed traffic away from a misconfigured JVM server (causing 20ms latency) within seconds, reducing P95/P99 degradation.
  • Operational Impact: Reduced slow-server alerts by 97% and engineering investigation time by 89%.

4. Significance

This paper presents a comprehensive, production-proven framework for enhancing the resilience of real-time OLAP systems. Its significance lies in:

1. Holistic Approach: It moves beyond raw performance optimization to address the operational realities of shared, multi-tenant, cloud-scale environments.
2. Sub-Millisecond Enforcement: The QWI mechanism achieves fine-grained resource isolation with negligible overhead, a feat difficult in Java-based OLAP engines due to opaque memory management.
3. Zero-Downtime Operations: The combination of MZ-aware placement and impact-free rebalancing allows for continuous cluster evolution (scaling, upgrades, maintenance) without service disruption, a critical requirement for modern data infrastructure.
4. General Applicability: While built for Apache Pinot, the principles of workload isolation, fault-domain aware placement, and adaptive routing are broadly applicable to other distributed OLAP and data processing systems.

The work demonstrates that strict SLAs can be maintained at scale not just by adding more hardware, but by implementing intelligent, adaptive control planes that manage resources, topology, and traffic dynamically.