Joint Hardware-Workload Co-Optimization for In-Memory Computing Accelerators

This paper proposes a joint hardware-workload co-optimization framework using an evolutionary algorithm to design generalized in-memory computing accelerators that significantly reduce the energy-delay-area product across multiple neural network workloads, overcoming the limitations of single-workload specialized designs.

The Gist

Imagine you are a chef trying to design the ultimate kitchen.

The Problem: The "One-Trick Pony" Kitchen

In the world of AI (Artificial Intelligence), computers need special "kitchens" called accelerators to cook up complex recipes (neural networks) quickly and without wasting energy.

Currently, most engineers design these kitchens for just one specific recipe.

• If you want to cook a giant turkey (a huge AI model), you build a massive oven with huge burners.
• If you want to cook a delicate soufflé (a small AI model), you build a tiny, precise stove.

The problem? Most real-world devices (like your phone or a self-driving car) need to cook many different recipes at once. If you use the "turkey kitchen" to make a soufflé, it's wasteful and slow. If you use the "soufflé kitchen" for a turkey, it burns out.

Existing methods try to solve this by either:

1. Designing a kitchen for the biggest recipe (which is overkill for small tasks).
2. Designing a kitchen for one specific recipe at a time (which doesn't work if you need to switch tasks).

The Solution: The "Universal Kitchen"

This paper introduces a new way to design a Universal Kitchen (a generalized In-Memory Computing accelerator) that can cook any recipe efficiently, from soufflés to turkeys, without wasting energy.

The authors call this "Joint Hardware-Workload Co-Optimization."

Here is how they did it, using some simple analogies:

1. The "Taste Test" Strategy (The Algorithm)

Instead of just guessing what the perfect kitchen looks like, they used a smart search engine called a Genetic Algorithm. Think of this like a reality TV cooking competition:

• The Contestants: They generate thousands of random kitchen designs (different sizes of ovens, different numbers of burners, different layouts).
• The Judges: They don't just test one dish. They cook four to nine different recipes on every single kitchen design.
• The Score: They give a score based on how fast it cooked, how much electricity it used, and how much counter space it took.
• The Evolution: The worst kitchens are thrown out. The best ones "mate" (combine their best features) to create new, better kitchens for the next round.

2. The "Smart Start" (Hamming Distance Sampling)

Usually, these competitions start with random kitchens. Sometimes, you get lucky and start with a great kitchen; other times, you start with a disaster. This leads to inconsistent results.

The authors added a clever twist: The Diversity Check.
Before the competition even starts, they look at all the random kitchens and pick the ones that are most different from each other.

• Analogy: Imagine picking a team of explorers. Instead of picking 5 people who all look the same, you pick one who is tall, one who is short, one who is an expert in deserts, and one who is an expert in snow. This ensures you cover all possibilities and don't get stuck in a "local trap" (like only exploring the desert when you needed to find a mountain).

3. The Four-Phase Cooking Process

They didn't just run the competition once. They ran it in four distinct phases, like a chef refining a dish:

• Phase 1 (Exploration): Throw everything at the wall. Try wild, crazy kitchen layouts to see what's possible.
• Phase 2 (Transition): Start narrowing it down. Keep the good ideas but mix them carefully.
• Phase 3 (Convergence): Focus on the top contenders. Make small, precise tweaks.
• Phase 4 (Fine-Tuning): The final polish. Adjust the knobs by a tiny fraction to get the perfect temperature.

The Results: Why It Matters

The results were impressive. By using this "Universal Kitchen" approach:

• Energy Savings: They reduced the energy cost (and time) by up to 95% compared to trying to force a "biggest recipe" kitchen to do everything.
• No Compromise: They proved you don't have to sacrifice performance to get a general-purpose machine. The "Universal Kitchen" was almost as good as a "Specialist Kitchen" for every single recipe.
• Flexibility: They tested this on two different types of "kitchen tools" (RRAM and SRAM memory) and even looked at how the cost of building the kitchen changes if you use different manufacturing technologies (like 7nm vs. 32nm chips).

The Big Picture

Think of this paper as a blueprint for building smart, adaptable AI chips. Instead of building a custom car for every single road trip, they figured out how to build one super-car that handles highways, dirt roads, and city streets equally well, saving money and fuel in the process.

This is a huge step forward for making AI faster, cheaper, and more energy-efficient for the devices we use every day.

Technical Summary

Here is a detailed technical summary of the paper "Joint Hardware-Workload Co-Optimization for In-Memory Computing Accelerators".

1. Problem Statement

The rapid growth of neural network models has necessitated energy-efficient hardware, leading to the development of In-Memory Computing (IMC) accelerators. However, existing software-hardware co-design frameworks suffer from a critical limitation: they typically optimize hardware parameters for a single specific workload (neural network model).

• The Gap: This results in highly specialized hardware that performs poorly when deployed on different models or applications.
• The Challenge: Practical deployment requires a generalized IMC platform capable of efficiently supporting a diverse set of workloads. Optimizing hardware parameters to support multiple workloads simultaneously is an open challenge, as it involves navigating a massive, non-convex, and discrete design space spanning device, circuit, architecture, and system levels.
• Current Limitations: Most state-of-the-art methods either optimize hardware for a single model or map diverse workloads onto fixed hardware without re-optimizing the hardware itself.

2. Methodology

The authors propose a Joint Hardware-Workload Co-Optimization Framework designed to identify generalized IMC architectures that minimize the performance gap between workload-specific and multi-workload designs.

A. Framework Overview

• Objective: To explore a vast hardware design space and identify optimal parameters that efficiently support a set of target neural network workloads.
• Evaluation Metric: The framework uses a joint score based on the Energy-Delay-Area Product (EDAP), calculated across all workloads in the target set.
• Simulation Engine: It utilizes the CIMLoop framework for fast, cycle-accurate hardware evaluation, supporting both RRAM (Resistive RAM) and SRAM-based architectures.
• Search Space: The framework optimizes parameters across four hierarchy levels:
  • Device: Bits per cell (RRAM).
  • Circuit: Operating voltage, cycle time (frequency).
  • Architecture: Crossbar dimensions (rows/columns), number of crossbars per tile, tiles per router.
  • System: Tile groups per chip, global buffer size, and technology node (CMOS).

B. Optimization Algorithm: Four-Phase Genetic Algorithm (GA)

The authors employ a modified Evolutionary Algorithm (specifically a Genetic Algorithm) to handle the discrete and high-dimensional search space. Key innovations include:

1. Hamming-Distance-Based Sampling:

   • Instead of purely random initialization, the algorithm first samples a large pool of candidates ($P_H = 1000$).
   • It then selects a diverse subset ($P_E = 500$) using a greedy algorithm based on Hamming distance. This ensures the initial population covers the design space broadly, preventing premature convergence to local minima.
   • For RRAM (weight-stationary), the sampling is constrained to designs that can fit the largest workload on-chip.

2. Four-Phase Search Strategy:
   The GA proceeds through four distinct phases, each with specific crossover ($P_c$) and mutation ($P_m$) probabilities and distribution indices ($\eta$):

   • Exploration: High mutation/crossover rates to broadly search the space.
   • Transition: Balances exploration and exploitation.
   • Convergence: Focuses on high-performing regions with finer variations.
   • Fine-tuning: Minimal mutation to refine the best solutions.

3. Aggregation Strategy:
   The objective function aggregates metrics across workloads. The paper compares "Max" (worst-case), "Mean," and "All" (product) aggregation, finding that Max-based aggregation (optimizing for the worst-case workload in the set) yields the best EDAP results and fastest convergence.

3. Key Contributions

1. Generalized Co-Optimization Framework: A novel approach that optimizes IMC hardware parameters to support multiple diverse workloads simultaneously, rather than a single model.
2. Advanced Optimization Algorithm: A four-phase GA with Hamming-distance-based sampling that significantly improves convergence stability and solution quality compared to traditional GAs.
3. Comprehensive Search Space: Unlike prior works limited to crossbar size or bits-per-cell, this framework optimizes across device, circuit, architecture, and system levels, including technology node selection.
4. Demonstrated Efficiency: Proof that the performance gap between generalized and workload-specific designs can be minimized without significant efficiency loss.

4. Key Results

The framework was evaluated on RRAM and SRAM architectures across various workloads (ResNet, VGG, MobileNet, ViT, GPT-2, etc.).

• EDAP Reduction: Compared to baseline methods (optimizing for a single workload or the largest workload only), the joint optimization achieved:
  • 76.2% EDAP reduction for a small set of 4 workloads.
  • 95.5% EDAP reduction for a large set of 9 workloads.
• Convergence Improvement: The proposed 4-phase GA with enhanced sampling reduced the EDAP score variance significantly (Std Dev dropped from 0.87 to 0.16) compared to traditional GAs, ensuring consistent results across runs.
• RRAM vs. SRAM:
  • RRAM: Achieved lower EDAP due to weight-stationary operation (no data movement for weights).
  • SRAM: Required weight swapping, leading to higher latency but offering flexibility for larger models.
• Ablation Studies: Joint optimization consistently outperformed sequential optimization (optimizing device $\to$ circuit $\to$ architecture step-by-step), which often got stuck in local minima or violated area constraints.
• Non-Idealities & Cost:
  • The framework successfully optimized designs considering RRAM non-idealities (noise, variability) and fabrication costs across different CMOS nodes (7nm to 90nm), revealing Pareto fronts for EDAP vs. Cost.
• Scalability: The method successfully scaled to include transformer-based models (ViT, GPT-2 Medium) and diverse architectures, proving adaptability to heterogeneous workloads.

5. Significance

This work addresses a critical bottleneck in the deployment of AI accelerators: the lack of hardware flexibility. By demonstrating that a single, generalized IMC architecture can be co-optimized to support diverse neural networks with minimal performance penalty, the paper provides a pathway for:

• Practical Deployment: Enabling edge devices and data centers to run multiple AI models on a single, efficient hardware platform.
• Design Automation: Providing a robust tool for hardware architects to navigate complex, multi-objective design spaces that were previously too large for manual or single-workload optimization.
• Future-Proofing: The framework's ability to incorporate technology nodes and cost metrics allows for the design of cost-effective, scalable accelerators tailored to specific manufacturing constraints.

In conclusion, the paper establishes that joint hardware-workload co-optimization is not only feasible but superior to traditional single-workload approaches, significantly narrowing the efficiency gap between specialized and generalized AI hardware.