🤖 machine learning

Harmonia: Enhancing Data Placement and Migration in Hybrid Storage Systems via Multi-Agent Reinforcement Learning

The paper introduces Harmonia, a multi-agent reinforcement learning framework that holistically optimizes interdependent data placement and migration policies in hybrid storage systems, achieving significant performance and cost improvements over prior single-policy approaches with minimal overhead.

Original authors: Rakesh Nadig, Vamanan Arulchelvan, Rahul Bera, Taha Shahroodi, Gagandeep Singh, Andreas Kakolyris, Ismail Emir Yuksel, Mohammad Sadrosadati, Jisung Park, Onur Mutlu

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rakesh Nadig, Vamanan Arulchelvan, Rahul Bera, Taha Shahroodi, Gagandeep Singh, Andreas Kakolyris, Ismail Emir Yuksel, Mohammad Sadrosadati, Jisung Park, Onur Mutlu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your computer's storage system is like a busy, high-end restaurant kitchen.

The Problem: A Chaotic Kitchen
In modern computing, applications (like AI models or databases) are like chefs who need ingredients (data) constantly. The kitchen has two types of counters:

The "Hot" Counter: A small, ultra-fast marble slab (like a high-speed SSD) where chefs can grab ingredients instantly. But it's tiny and fills up quickly.
The "Cold" Counter: A massive, slow wooden table (like a hard drive) that holds everything but takes forever to walk to.

The goal is to keep the most-used ingredients on the fast counter so the chefs don't waste time walking to the slow table.

For years, computer scientists tried to solve this with two separate managers:

The Placement Manager: Decides where to put a new ingredient when it first arrives.
The Migration Manager: Decides which old ingredients to move off the fast counter to make room, or which slow ones to bring up.

The Flaw: These two managers worked in isolation. The Placement Manager might shove a new item onto the fast counter, not realizing the Migration Manager was about to kick out a different item that was actually more important. Or, the Migration Manager might move things around while the Placement Manager was trying to work, causing a traffic jam. It was like having two chefs trying to organize the same shelf without talking to each other—they kept stepping on each other's toes, slowing down the whole kitchen.

The Solution: Harmonia (The Harmony Team)
The researchers created a new system called Harmonia. Instead of one boss or two silent managers, they introduced a team of two AI assistants (called "Multi-Agent Reinforcement Learning") that talk to each other.

Think of them as two specialized sous-chefs who have learned to dance together:

The Placement Sous-Chef: This agent is super fast. When a new order comes in, it instantly decides: "Should I put this on the fast marble counter or the slow wooden table?" It learns to be smart about saving space on the fast counter for the most critical items.
The Migration Sous-Chef: This agent works in the background. While the kitchen is quiet for a split second, it looks at what's sitting on the counters. It asks: "Is this item on the fast counter being ignored? Is there a hot item on the slow table that needs to move up?" It quietly swaps things around to keep the fast counter optimized for the future.

How They Work Together
The magic of Harmonia is that these two agents don't just act alone; they coordinate.

The Placement agent knows that if it fills up the fast counter with junk, the Migration agent will have to work overtime to clean it up.
The Migration agent knows that if it clears space on the fast counter, the Placement agent can do a better job with new orders.

They use a "reward system" (like a scorecard). If the kitchen runs smoothly and orders are fast, both get a high score. If they cause a traffic jam, they get a low score. Over time, they learn the perfect rhythm of working together without getting in each other's way.

The Results
The researchers tested this on real computer systems with different types of storage (from fast SSDs to slow hard drives).

Speed: Harmonia made the kitchen run 29% to 44% faster than the best previous methods.
Efficiency: It reduced the time chefs (applications) spent waiting for ingredients.
Cost: It did all this with almost no extra cost. The "brain" of the system is so small it fits in a tiny corner of the computer's memory (about the size of a small photo), and it makes decisions in the blink of an eye (240 nanoseconds).

Why It Matters
Before this, trying to optimize both placing new data and moving old data at the same time was like trying to juggle while riding a unicycle. The researchers showed that by using two specialized AI agents that coordinate like a well-rehearsed dance team, you can get the best of both worlds: a fast, efficient storage system that adapts to whatever the computer is doing, whether it's training an AI, running a database, or processing a genome.

In short, Harmonia turns a chaotic kitchen with arguing managers into a synchronized, high-speed operation where the right ingredients are always in the right place at the right time.

Technical Summary: Harmonia

Problem Statement

Modern High-Performance Computing (HPC) environments rely on Hybrid Storage Systems (HSS) that integrate heterogeneous devices (e.g., fast SSDs, slow HDDs, PMEM) to balance performance, capacity, and cost. The performance of an HSS is critically dependent on two interdependent data-management policies:

Data Placement: Determining the optimal device for incoming write or allocation requests.
Data Migration: Dynamically reorganizing stored data (prefetching hot data, evicting cold data) to sustain performance as workloads and system conditions evolve.

Prior research has largely optimized these policies in isolation. The paper argues that optimizing placement without considering migration (or vice versa) leads to suboptimal performance because the policies are tightly coupled. For instance, a static migration policy cannot sustain the benefits of an optimized placement policy if workload access patterns change. Furthermore, naive combinations of state-of-the-art placement and migration techniques often result in conflicting decisions, unnecessary data movement, and wasted bandwidth. Existing single-agent Reinforcement Learning (RL) approaches also struggle because placement and migration differ fundamentally in their objectives, data inputs, and execution timescales (placement is on the I/O critical path, while migration is asynchronous), leading to task interference.

Methodology: Harmonia

To address these limitations, the authors propose Harmonia, a holistic data-management technique based on Multi-Agent Reinforcement Learning (MARL). Harmonia employs two lightweight, autonomous RL agents that coordinate to optimize the HSS:

1. Agent Architecture

Data-Placement Agent: Operates on the critical path of I/O handling. It selects the most suitable storage device for incoming write requests. It builds upon the state-of-the-art technique Sibyl but incorporates new migration-related state features and a reward structure that encourages effective utilization of fast devices without causing contention.
Data-Migration Agent: Operates asynchronously in the background. It monitors previously placed pages, identifies migration candidates, and selects target devices. It aims to maximize long-term HSS performance by proactively moving data based on predicted access patterns.

2. Coordination Mechanism

The agents coordinate through a shared state representation and a carefully designed reward structure, rather than a joint action space:

Shared State: Both agents observe a 32-bit state vector containing seven features: request type, request size, access interval, access frequency, fast-device free capacity, current device location, and migration interval.
Reward Structure:
- The Placement Agent receives an immediate reward inversely proportional to the latency of the current I/O request.
- The Migration Agent receives a delayed reward based on the average placement latency of the subsequent $n$ I/O requests (set to 50). This encourages migration decisions that improve future placement efficiency.
- A penalty is applied to the migration agent to prevent "ping-pong" migrations of the same page.
Conflict Mitigation: The migration agent ignores recently placed or migrated pages and only acts during system idle periods or low-utilization windows to avoid interfering with critical I/O requests.

3. Implementation Details

RL Algorithm: Both agents use C51 (a categorical Deep Q-Network) to learn the distribution of Q-values.
Threading: Each agent has separate training and inference threads. The inference thread runs on the critical path (for placement) or in the background (for migration), ensuring low latency.
Overhead: The system is designed for low overhead, with inference latency of ~240 ns and a total storage footprint of ~206 KiB in DRAM for both agents combined.

Key Contributions

Empirical Demonstration of Limitations: The authors show that prior HSS data-management approaches (heuristic and ML-based) optimize placement or migration in isolation. They demonstrate that even the best prior approach (Sibyl) incurs significantly higher latency (100.4% higher on performance-optimized HSS) compared to an ideal "Oracle" policy.
Extended Baselines: The paper constructs four extended techniques (e.g., combining Sibyl with K-SVM or RNN-HSS) to prove that naive combinations of state-of-the-art policies perform worse than their constituent parts due to policy conflicts.
Harmonia Proposal: The introduction of a multi-agent RL framework that co-optimizes placement and migration using two specialized, coordinated agents.
Real-System Evaluation: A rigorous evaluation on a real Linux system with up to four heterogeneous storage devices and 25 diverse data-intensive workloads (including ML, databases, and graph processing).

Evaluation Results

Harmonia was evaluated against eight baselines (including Sibyl, CDE, K-SVM, RNN-HSS, and single-agent RL variants) across various HSS configurations (Dual, Tri, and Quad-HSS) and workloads.

Performance Gains:
- On a performance-optimized Dual-HSS, Harmonia reduces average I/O request latency by 29.3% compared to the best prior approach (Sibyl).
- On a cost-optimized Dual-HSS, the improvement is 44.8%.
- On Tri-HSS and Quad-HSS configurations, Harmonia outperforms Sibyl by 38.9% and 39.2%, respectively.
- In multi-programmed workloads, improvements scale with concurrency, reaching up to 39.4% on eight concurrent workloads.
Comparison to Oracle: Harmonia achieves performance close to the theoretical Oracle (which has perfect knowledge of future access patterns), with latency only 41.0% higher than Oracle on performance-optimized HSS.
Tail Latency: Harmonia significantly reduces tail latency (99th and 99.99th percentiles) by up to 32.3% compared to Sibyl, primarily by avoiding bulk migrations on the critical path.
Overhead: The technique incurs minimal overhead: 240 ns for inference and 206 KiB of DRAM storage.
Device Lifetime: Harmonia slightly increases Write Amplification (WA) compared to some baselines (e.g., 1.44 vs. 1.38 for Sibyl on performance-optimized HSS) due to proactive prefetching of hot data, but this trade-off is accepted for the significant latency gains.

Significance and Claims

The paper claims that Harmonia is the first multi-agent RL-based holistic data-management technique that jointly optimizes data placement and migration in hybrid storage systems. Its significance lies in:

Holistic Optimization: Moving beyond the isolation of placement and migration policies to address their interdependence.
Decoupled Multi-Agent Design: Demonstrating that using heterogeneous agents with different timescales and objectives, coordinated via rewards, is superior to single-agent formulations or naive combinations of existing techniques.
Practicality: Proving that such complex learning-based management can be implemented with low latency and storage overhead, making it viable for latency-sensitive HPC applications.
Extensibility: Showing that the framework scales to HSS configurations with up to four devices and generalizes well to unseen workloads and emerging technologies like Persistent Memory (PMEM).

The authors conclude that Harmonia paves the way for self-optimizing storage systems that can adaptively manage data across heterogeneous devices under dynamic workload conditions.