Technical Summary: Misleading Microbenchmarks on the Java Virtual Machines

Problem Statement

Developers frequently rely on microbenchmarks, particularly using the Java Microbenchmark Harness (JMH), to evaluate and optimize specific methods or classes. While JMH addresses common pitfalls regarding measurement noise and statistical rigor, this paper argues that it cannot resolve the fundamental issue of execution context.

The core problem is that microbenchmarks execute code in a "sterile environment." In this setting, a method is invoked repeatedly from a single call site with fixed parameters, isolated from other application code. On the Java Virtual Machine (JVM), which utilizes tiered, profile-driven Just-In-Time (JIT) compilation, this isolation leads to the collection of unrealistic execution profiles.

Because the JVM's speculative optimizations (such as inlining, branch prediction, and type specialization) depend heavily on runtime profiles (e.g., branch probabilities, receiver types, and code hotness), a sterile environment causes the JIT compiler to generate code that is over-specialized for the specific benchmark conditions. These aggressive optimizations often do not occur in real-world applications where code is executed in a "context-rich" environment with heterogeneous inputs and multiple call sites. Consequently, developers may derive misleading conclusions, believing an implementation is faster when it is actually slower in realistic scenarios, or vice versa.

Methodology

The authors investigate this phenomenon through three distinct case studies, comparing performance results obtained via two primary execution strategies:

1. Do-Nothing Strategy: The standard microbenchmarking approach where the benchmark runs in isolation with fixed inputs, mimicking a sterile environment.
2. Manual-Pollute Strategy: A proposed mitigation where the benchmark setup phase (@Setup) exercises the target method with a diverse set of input parameters to cover various code paths before measurement begins. This aims to "pollute" the profile with realistic diversity, preventing over-specialization.
3. Realistic Traces (Validation): In the first case study, the authors employ a record-and-replay approach using traces from the DaCapo-Chopin and Renaissance benchmark suites to validate the representativeness of the Manual-Pollute strategy.

The experiments were conducted on JDK-24 using JMH version 1.37 on an Intel Xeon Gold 6326 CPU, ensuring hyper-threading and turbo boost were disabled for stability.

Key Contributions and Case Studies

1. Optimizing a Single Function: hashCode()

• Scenario: The authors compared a standard polynomial rolling hash implementation (baseline) against an alternative implementation using a switch statement optimized for short arrays (up to 32 elements).
• Findings: Under the Do-Nothing strategy, the switch-based implementation appeared consistently faster (geometric mean speedup of 1.10x). The JVM optimized the code assuming the array length was constant, effectively removing branches.
• Reality: Under the Manual-Pollute strategy and Realistic Traces, the alternative implementation was slower (geometric mean speedup of 0.91x) because the JIT compiler could no longer specialize for a single length and had to handle diverse inputs.
• Conclusion: Sterile environments can create artificial performance advantages for specialized code that vanishes in realistic usage.

2. Benchmarking Libraries: The Stream API

• Scenario: The authors evaluated TPC-H queries using the Java Stream API. They compared performance when the Stream API was used in isolation versus when it was pre-exercised with a representative set of operations (map, filter, collect) during the setup phase.
• Findings: The Do-Nothing strategy yielded results showing a 1.08x geometric mean speedup over the Manual-Pollute strategy (peaking at 1.41x).
• Root Cause: In the sterile environment, the JIT compiler speculatively inlined specific lambda functions into the Stream API methods because it only observed one type of callback. In the polluted environment, the compiler observed multiple different lambdas, preventing this specific global specialization and resulting in more general, albeit slower, code.
• Conclusion: Even complex library code suffers from over-specialization in sterile benchmarks, leading to unrealistic performance gains.

3. Benchmarking JVM-Initialized Classes: Collections

• Scenario: The authors compared standard Java Collection classes (e.g., HashMap, HashSet from java.util) against code clones of these classes placed in a different package.
• Findings: The code clones consistently outperformed the standard JCL classes (geometric mean speedups ranging from 1.01x to 1.34x) when using the Do-Nothing strategy.
• Root Cause: Standard JCL classes are exercised during JVM initialization (before the benchmark starts), collecting profiles that may include "unlucky" or diverse paths unrelated to the benchmark. The code clones, however, are only exercised during the benchmark in a sterile environment, allowing them to be optimized specifically for the test.
• Conclusion: Comparing user-defined alternatives directly against JCL classes introduces a systematic bias where the JCL classes appear slower due to initialization profiling, while the alternatives benefit from sterile optimization.

Results and Recommendations

The paper demonstrates that following existing JMH guidelines (warmup, isolation, statistical rigor) is insufficient to prevent misleading results caused by profile-driven JIT optimizations.

Key Recommendations:

• Avoid Fixed Inputs: Do not rely solely on fixed parameters in microbenchmarks.
• Exercise Diverse Paths: Use the @Setup phase to exercise the target code with a variety of inputs (Manual-Pollute strategy) to ensure the JIT compiler generates code that handles diverse scenarios, not just the benchmark's specific case.
• Fair Comparison of Libraries: When comparing alternatives to JCL classes, either:
  1. Copy the JCL classes to ensure both versions run in the same sterile environment (though this still suffers from the sterile environment problem).
  2. Use Java patch mechanisms to replace JCL classes with user implementations, ensuring both are exercised during initialization and benchmarked under realistic conditions.

Significance

The paper claims that the "sterile environment" is an inherent limitation of microbenchmarking that leads to unrealistic compiler optimizations. The significance of this work lies in its demonstration that even well-constructed microbenchmarks can lead developers to make wrong optimization choices and incorrect conclusions about the relative performance of implementations.

The authors emphasize that performance engineering must account for high-level platform behaviors (like JIT profiling) that emerge only in realistic contexts, not just low-level factors like memory layout. By highlighting the gap between sterile benchmark results and realistic application performance, the paper urges the community to adopt execution strategies that better reflect the diversity of real-world workloads.