Technical Summary: A Modern Large-Scale Memory Characterization Laboratory

Problem Statement

As DRAM technology scales to smaller nodes, it faces critical challenges regarding data integrity, latency, and robustness. These issues fundamentally impact system performance, security, and energy efficiency. While researchers rely on empirical characterization to understand these phenomena, access to large-scale, flexible testing infrastructure is often limited. Manual experimentation across diverse DRAM chips (varying by manufacturer, die revision, and density) is labor-intensive, prone to hardware damage from repeated module swapping, and insufficient for gathering the statistical rigor required to discover subtle, widespread vulnerabilities or novel functionalities. Furthermore, detailed empirical power consumption data for newer standards (such as DDR4 and HBM2) remains scarce, hindering the development of accurate power models.

Methodology

The authors describe the evolution and expansion of DRAM Bender, a versatile, FPGA-based infrastructure originally developed by the SAFARI Research Group. The paper details the establishment of a large-scale characterization laboratory comprising over 100 DRAM Bender setups. The methodology focuses on three primary technical updates to the infrastructure:

1. Power Measurement Integration:

   • DDR4: A custom riser board was designed to sit between the DIMM slot and the module. This board intercepts power rails and routes them through on-board components to provide accurate, real-time power readings, transmitted to a host via USB.
   • HBM2: Power measurement for High Bandwidth Memory stacks is achieved by integrating the AMD Card Management Subsystem (CMS) into Alveo U55C FPGA boards. The DRAM Bender software stack fetches instantaneous, maximum, and average current values via the PCIe interface from the satellite controller.

2. High Bandwidth Memory (HBM2) Support:

   • The infrastructure was updated to support HBM2 stacks on Bittware XUPVVH and Alveo U50 FPGA boards.
   • This setup allows precise control of HBM2 command timings at a granularity of 1.67 ns (600 MHz interface clock).
   • The system supports 4 GiB HBM2 stacks with 8 channels, 2 pseudo-channels, 16 banks, 16K rows, and 1 KiB row sizes.

3. Large-Scale Job Scheduling:

   • To manage the 100+ setups and eliminate manual module replacement, the authors integrated DRAM Bender with Slurm, a widely-used open-source workload manager.
   • Each DRAM Bender setup is modeled as a Slurm node. The attributes of the installed DRAM chip (manufacturer, revision, standard) and supplementary hardware (temperature controllers, power boards) are exposed as Slurm features.
   • Researchers submit jobs tagged with required features; Slurm dispatches these to compatible, idle setups, enabling static allocation of diverse DRAM chips and fair resolution of resource contention.

Key Contributions

The paper introduces the following specific contributions to the memory research community:

• Infrastructure Expansion: The first public introduction of DRAM Bender's updated capabilities, specifically its support for HBM2 and its ability to perform fine-grained power measurements on both DDR4 and HBM2.
• Large-Scale Laboratory: The deployment of a 100+ node characterization laboratory that enables concurrent testing of diverse DRAM chips without manual intervention, significantly accelerating the discovery of new memory characteristics.
• Open-Source Tools: The integration of DRAM Bender with Slurm for automated job scheduling, which the authors intend to release publicly to allow other groups to operate similar large-scale laboratories.
• Educational Resources: Continued support for the community through tutorial series (Ramulator and DRAM Bender) at major conferences (ASPLOS, ISCA, ICS) and undergraduate courses at ETH Zürich, providing hands-on experience with real DRAM characterization.

Results and Capabilities

The paper does not present new experimental data as its primary result but rather highlights the enabling nature of the infrastructure. It summarizes the capabilities of the updated DRAM Bender by citing prior and ongoing work that utilizes or will utilize this infrastructure:

• Robustness Discoveries: The infrastructure has enabled the discovery and study of phenomena such as RowHammer, RowPress, Variable Read Disturbance, ColumnDisturb, and the DejaVu effect. It has facilitated the observation of bitflips under standard operating conditions and the analysis of sensitivities to temperature and access patterns.
• Novel Functionality: The ability to issue arbitrary commands with arbitrary delays has allowed researchers to uncover previously unknown capabilities in commercial off-the-shelf (COTS) DRAM, including Processing-in-DRAM (PUD) operations (e.g., RowClone, bitwise majority, Boolean logic), True Random Number Generation (TRNG) via timing violations, and Physical Unclonable Functions (PUFs) based on latency variations.
• HBM2 Validation: The HBM2 support has been used to demonstrate for the first time that read disturbance phenomena (RowHammer and RowPress) are prevalent in real HBM2 stacks.
• Power Modeling: The new power measurement capabilities aim to provide empirical data for DDR4 and HBM2 to develop better power models, following the rigorous calibration previously done for DDR3.

Significance and Claims

The paper claims that this large-scale laboratory and the updated DRAM Bender infrastructure are critical for overcoming the "huge memory bottleneck" plaguing modern computing systems. By enabling the community to:

1. Discover new problems: Uncover fundamental operational characteristics and vulnerabilities in modern DRAM.
2. Solve scaling issues: Develop mechanisms to improve memory performance, robustness, security, and energy efficiency.
3. Accelerate research: Allow for the concurrent testing of diverse hypotheses and the in-depth quantitative analysis of previously known issues that were previously discarded due to resource constraints.

The authors position this work as a foundational step to help the research community move beyond theoretical models to empirical validation, ultimately leading to more secure and efficient memory systems. The paper emphasizes that the infrastructure is designed to be accessible to both software and hardware developers, fostering a broader segment of the community to conduct rigorous experimental DRAM research.