Technical Summary: Hermes

Problem Statement

Long-latency load requests that miss the on-chip cache hierarchy and access off-chip main memory remain a primary bottleneck for modern high-performance out-of-order (OOO) processors. These requests often stall the processor core by blocking instruction retirement from the Reorder Buffer (ROB).

The authors identify two critical trends in current processor designs that limit performance despite existing optimizations:

1. Limited Prefetcher Coverage: Even state-of-the-art hardware prefetchers (e.g., Pythia) can only predict approximately half of the off-chip load requests across a wide range of workloads. The remaining unpredictable loads continue to cause significant stalls.
2. Increasing On-Chip Cache Latency: As on-chip caches grow in size and complexity to accommodate larger data footprints, the latency required to traverse the on-chip cache hierarchy (L1, L2, LLC) to determine that a request is a miss has increased significantly. A substantial fraction of the total latency of an off-chip load is spent merely accessing the on-chip hierarchy to confirm the miss, rather than waiting for the main memory response.

Methodology

The paper proposes Hermes, a technique designed to accelerate off-chip load requests by removing the on-chip cache access latency from their critical path.

Core Mechanism

Hermes operates on the principle of speculative off-chip fetching:

1. Prediction: For every load request, a predictor determines if the request is likely to miss the on-chip hierarchy and go off-chip.
2. Speculative Fetch: If a load is predicted to go off-chip, Hermes issues a speculative request (a "Hermes request") directly to the main memory controller as soon as the physical address is generated.
3. Concurrent Execution: This speculative request proceeds concurrently with the regular load request, which still traverses the standard on-chip cache hierarchy.
4. Latency Hiding: If the prediction is correct, the regular load eventually misses the Last-Level Cache (LLC) and waits for the ongoing speculative request to complete. This effectively hides the on-chip cache traversal latency under the shadow of the main memory latency.
5. Coherence: If the prediction is incorrect (the load actually hits in the cache), the speculative request is dropped, and no data is filled into the cache, ensuring cache coherence is maintained without complex recovery mechanisms.

POPET: The Predictor

To enable Hermes, the authors introduce POPET (Perceptron-based Off-chip load Prediction Technique), a lightweight predictor that learns to identify off-chip loads using multiple program features.

• Algorithm: POPET utilizes a hashed perceptron learning model. It aggregates weights from various program features to make a binary classification (off-chip vs. on-chip).
• Features: The predictor uses a combination of features including:
  • XOR of Program Counter (PC) and cache line offset.
  • XOR of PC and byte offset.
  • PC plus a "first access" hint (tracking recent page touches).
  • Cache line offset plus "first access."
  • A history of the last four load PCs.
• Training: POPET is trained online using the actual outcome of load requests (whether they went off-chip) to update feature weights.
• Overhead: POPET requires only 4 KB of storage per core (3.2 KB for weight tables and 0.8 KB for Load Queue metadata), avoiding the impractical metadata overheads of prior address-tracking approaches.

Key Contributions

1. Identification of Opportunities: The paper highlights that a significant fraction of off-chip loads remain unpredictable by current prefetchers and that a growing portion of off-chip latency is consumed by on-chip cache traversal.
2. Hermes Technique: The introduction of a mechanism that speculatively fetches data from main memory for predicted off-chip loads while concurrently accessing the cache hierarchy, thereby eliminating on-chip latency from the critical path.
3. POPET Predictor: The design of a new, low-overhead perceptron-based predictor that achieves high accuracy and coverage for off-chip load prediction without requiring large metadata structures or OS support.
4. Comprehensive Evaluation: Extensive evaluation across 110 single-core and 220 multi-core workloads (SPEC CPU2006/2017, PARSEC, Ligra, CVP) and various system configurations (varying core counts, memory bandwidth, and prefetchers).
5. Open Source: The release of Hermes source code, traces, and scripts to reproduce results.

Results

The evaluation demonstrates that Hermes provides consistent performance improvements over state-of-the-art baselines:

• Prediction Accuracy: POPET achieves an average of 77.1% accuracy and 74.3% coverage for off-chip loads. This is significantly higher than the prior best-accuracy predictor (HMP), which achieved 47% accuracy and 22.3% coverage.
• Performance Gains:
  • Single-Core: Hermes combined with the Pythia prefetcher improves performance by an average of 5.4% (up to 23.4%) over Pythia alone.
  • Multi-Core (8-core): Improvements average 5.1% (up to 25.7%).
  • Bandwidth-Constrained: In bandwidth-constrained configurations, Hermes improves performance by 6.2% (up to 32.2%) over Pythia, outperforming Pythia alone in these scenarios due to lower memory bandwidth overhead.
• Efficiency: Hermes incurs a modest 0.5% increase in main memory requests for every 1% performance gain, whereas the Pythia prefetcher incurs a 2% increase in requests for the same gain.
• Robustness: Hermes consistently improves performance when combined with various high-performance prefetchers (Bingo, SPP, MLOP, SMS) and remains effective even with pessimistic estimates of on-chip interconnect latency (up to 24 cycles).
• Power and Storage: Hermes adds only 3.6% dynamic power overhead (compared to 8.7% for Pythia) and requires only 4 KB of storage per core, compared to 25.5 KB for Pythia.

Significance and Claims

The authors claim that Hermes represents a significant step forward in handling long-latency memory accesses by fundamentally changing the critical path of off-chip loads. Unlike traditional prefetchers that attempt to hide latency by fetching data before it is needed, Hermes reduces the latency of the load itself by bypassing the on-chip hierarchy for predicted misses.

The paper emphasizes that as on-chip caches continue to grow in size and complexity, the latency penalty for traversing them to confirm a miss will only increase. Hermes addresses this specific trend by decoupling the decision to fetch from main memory from the latency of the on-chip cache hierarchy. The authors posit that the key observation—that off-chip loads can be predicted with high accuracy using lightweight perceptron learning—opens the door for future memory system optimizations that further reduce the impact of long-latency requests.

The work is presented as a practical solution that does not require OS support or impractical metadata storage, making it a viable candidate for adoption in commercial processors. The authors conclude that Hermes provides a "significant and consistent performance improvement" across a wide range of workloads and system configurations.