An Updated Assessment of Reinforcement Learning for Macro Placement

This paper presents an updated and rigorous assessment of Google's deep reinforcement learning approach (Circuit Training) for macro placement by introducing stronger baselines, new sub-10nm benchmarks, and commercial-grade evaluations to address reproducibility challenges and identify remaining open questions regarding scalability and pre-training methodologies.

The Gist

Imagine you are trying to pack a massive, incredibly complex moving truck. The truck is a computer chip, and the items you need to pack are huge, heavy furniture pieces (called "macros") like memory banks and processor cores, mixed with millions of tiny Lego bricks (standard cells).

Your goal is to arrange these items so the truck is as small as possible, the items don't rattle around too much (low power), and you can easily reach everything without tripping over wires (high performance).

For decades, humans and traditional computer algorithms have been the best at solving this packing puzzle. But in 2021, Google announced a new "AI Robot" (called AlphaChip or Circuit Training) that claimed it could pack this truck better than any human, in less than six hours, using deep learning.

This paper is a reality check. A team of researchers decided to test Google's claim again, but this time with a much stricter rulebook, better tools, and a fairer playing field. Here is what they found, explained simply:

1. The "Black Box" Problem

When Google first announced their AI, they said, "Here is our amazing result!" but they didn't give everyone the exact recipe or the exact ingredients. It was like a magician showing a trick but hiding the secret move.

• The Issue: Other scientists couldn't fully copy the experiment because the code and data were incomplete or "black boxed."
• The Fix: This new paper built a completely open, transparent lab. They recreated the environment, fixed the missing pieces, and made sure everyone could see exactly how the sausage was made.

2. The "Old Dog" vs. The "New Puppy"

The researchers compared three main contenders:

• The Human Expert: A master packer who has done this for years.
• The "Old Dog" (Simulated Annealing): A classic, math-based algorithm that has been around since the 1980s. The researchers upgraded this old dog with a new "super collar" (a technique called "Go-With-The-Winners" and multi-threading) to make it run faster and smarter.
• The "New Puppy" (Google's AI): The latest version of Google's reinforcement learning robot.

The Result:
Surprisingly, the upgraded "Old Dog" (Simulated Annealing) and the Human Experts consistently packed the truck better than the AI.

• Better Packing: The traditional methods created layouts with less wire length (fewer tangled wires) and better timing.
• Cheaper & Faster: The AI required a massive amount of computing power (like a supercomputer farm) and took a long time to train. The traditional methods did the same job using a fraction of the energy and time.

3. The "Practice Makes Perfect" Myth

Google suggested that their AI gets better if you "pre-train" it on other chips first (like a student studying math before taking a physics exam).

• The Study: The researchers tried this pre-training method.
• The Finding: While pre-training helped the AI converge on some problems, it didn't magically make it beat the traditional methods. In fact, when the chips got really huge (scaled up), the AI often got confused and failed to find a good solution, whereas the traditional methods kept working steadily.

4. The "Fake Score" vs. The "Real Score"

This is the most critical insight.

• The Proxy Score: The AI was trained to minimize a "proxy score" (a simulation of how good the chip is). It was like a student trying to get a high score on a practice test.
• The Real Score: The researchers ran the final designs through a commercial "Place-and-Route" tool (the real factory simulator) to see the actual performance, power, and area.
• The Disconnect: They found that a low "proxy score" (a good practice test result) did not guarantee a good real-world result. The AI was optimizing for the wrong things. It was like a student who memorized the practice test answers perfectly but failed the actual exam because the questions were slightly different.

5. The "Stability" Issue

The researchers found that the AI is a bit of a "gamble."

• If you run the AI twice with the same settings, you might get two very different results. Sometimes it works great; sometimes it fails completely.
• The traditional methods are like a reliable clock: if you set them up the same way, they give you the exact same high-quality result every time.

The Big Takeaway

This paper isn't saying AI is useless. It's saying that hype doesn't equal reality.

• Reproducibility Matters: Science needs to be open. If you can't copy the experiment, you can't trust the result.
• Don't Ignore the Classics: Just because a new, flashy technology (AI) arrives, it doesn't mean the old, well-understood tools (Simulated Annealing) are obsolete. In fact, when optimized correctly, the old tools are still winning.
• Measure What Matters: You can't just optimize for a simulation score; you have to optimize for the final, real-world product.

In short: The AI robot is a promising student, but the veteran human packer and the upgraded classic algorithm are still the ones getting the job done best, faster, and cheaper. The research community needs to be careful not to get swept up in the hype before the data is truly solid.

Technical Summary

1. Problem Statement

Macro placement is a critical, NP-hard step in VLSI physical design where large circuit blocks (macros) must be positioned on a chip layout. The goal is to optimize multiple conflicting objectives: wirelength, area utilization, timing closure, power consumption, and routing congestion.

In 2021, Google published a paper in Nature claiming that a Deep Reinforcement Learning (RL) approach (Circuit Training) could generate chip floorplans superior to human experts and traditional algorithms in under six hours. However, subsequent attempts to reproduce these results faced significant hurdles due to:

• Lack of full data and code availability.
• Discrepancies between the published Nature claims and the open-source "Circuit Training" (CT) repository.
• Questions regarding the sufficiency of compute resources, training convergence, and the validity of the "proxy cost" function used as the RL reward.

This paper aims to provide a rigorous, reproducible, and evidence-based assessment of the RL approach (specifically the updated "AlphaChip" version) against strong classical baselines and human experts.

2. Methodology

The authors conducted a comprehensive re-evaluation using a transparent, open-source framework.

A. Benchmarking and Testcases

• Sub-10nm Enablement: The study moved beyond older nodes to sub-10nm technologies.
  • Converted Google's TSMC 7nm Ariane protobuf data to standard LEF/DEF formats.
  • Created scaled variants (x2 and x4) of the Ariane design to test scalability (up to 532 macros).
  • Ported testcases to the open-source ASAP7 7nm PDK and commercial GlobalFoundries 12nm (GF12) flows.
• Evaluation Metrics: Unlike previous studies that relied on "proxy costs" (wirelength, density, congestion), this study used a commercial Place-and-Route (P&R) flow (Cadence Genus and Innovus) to generate "true" post-route metrics: Power, Performance (Timing/WNS/TNS), and Area (PPA).

B. Algorithms Compared

1. Circuit Training (CT) Variants:
   • CT-Scratch: Training the latest AlphaChip model from scratch.
   • CT-AC: Fine-tuning Google's pre-trained "AlphaChip" checkpoint (released Aug 2024).
   • CT-Ours: Fine-tuning a model pre-trained on specific slices of the target design.
2. Simulated Annealing (SA) Baseline:
   • The authors developed a stronger SA baseline incorporating a multi-threaded "Go-With-The-Winners" (GWTW) metaheuristic.
   • This approach uses 80 parallel threads with periodic synchronization to clone the best solutions, significantly improving efficiency over the 320 independent workers used in prior work.
3. Other Baselines:
   • RePlAce: An academic electrostatic-based placer.
   • CMP: A commercial concurrent macro placer (Cadence Innovus).
   • Human Experts: Placements contributed by experts from IBM, ETH Zurich, and UCSD.

C. Experimental Protocol

• Reproducibility: All scripts, data, and Docker/Singularity images are public.
• Resource Allocation: The authors ensured sufficient compute resources (8x NVIDIA V100 GPUs, 256 collect jobs) and extended training iterations (400 total) to address previous criticisms of under-training.
• Stability Analysis: Multiple runs with different random seeds were conducted to assess stochasticity in both CT and SA.

3. Key Contributions

1. Improved SA Baseline: The new GWTW-based SA implementation achieved up to 26% better proxy cost within the same runtime while using only 1/4 of the CPU resources compared to previous SA implementations.
2. Sub-10nm Benchmarks: The release of LEF/DEF versions of Google's 7nm Ariane design and scaled variants, enabling fair evaluation in modern technology nodes.
3. Pre-Training Analysis: A detailed study of the "AlphaChip" pre-training methodology, revealing limitations in scalability and convergence when applied to larger or more diverse datasets.
4. Correlation Study: An analysis showing a weak correlation between the RL "proxy cost" (the reward function) and the actual post-route PPA metrics (the "ground truth").

4. Key Results

A. Performance Comparison

• SA vs. RL: The improved Simulated Annealing (SA) consistently outperformed both the "from-scratch" (CT-Scratch) and "fine-tuned" (CT-AC) RL models across most metrics.
  • Wirelength: SA achieved better routed wirelength in 7 out of 9 cases compared to CT-AC.
  • Proxy Cost: SA dominated in 6 out of 9 cases.
  • Human Experts: For large, complex designs (e.g., BlackParrot, MemPoolGroup), human experts often outperformed CT-AC in final PPA metrics.
• Resource Efficiency: SA required significantly fewer resources. For example, on the BlackParrot design, SA used ~896 CPU-hours, whereas CT-Scratch and CT-AC required ~19,000 and ~21,500 CPU-hours respectively (factoring in GPU-to-CPU equivalence).

B. Stability and Scalability

• Stochasticity: CT training showed high instability. Identical runs on the same machine with the same seed could result in either convergence or divergence.
• Scalability Issues: When scaled to the CT-Ariane-X4 design (532 macros), training from scratch failed to converge (diverged). While pre-training helped CT-AC converge, it still underperformed SA in terms of wirelength and proxy cost.
• Pre-Training Limitations: The study found that increasing the diversity of pre-training datasets often led to divergence, and pre-training on smaller slices did not guarantee success on larger blocks.

C. The Proxy Cost Problem

The paper highlights a critical misalignment: the proxy cost (wirelength, density, congestion) optimized by the RL agent does not strongly correlate with the final post-route PPA metrics (Power, Timing, Area).

• Table IX shows Kendall rank correlation coefficients near 0 for many metrics, suggesting that optimizing the proxy cost does not reliably lead to a better final chip design.

5. Significance and Conclusion

This paper serves as a critical "reality check" for the application of Deep RL in EDA:

• Reproducibility Crisis: It underscores the scientific necessity of open data, code, and rigorous baselines. High-profile claims in Nature were not fully reproducible without significant effort and access to proprietary tools.
• Classical Heuristics vs. AI: The results suggest that for macro placement, carefully optimized classical heuristics (like Simulated Annealing) remain superior to current RL approaches in terms of solution quality, stability, and resource efficiency.
• Methodological Flaw: The reliance on a proxy cost function that poorly correlates with final chip metrics is identified as a fundamental flaw in the current RL formulation for this problem.
• Future Direction: The authors call for "frictionless reproducibility" in the EDA community, emphasizing that AI/ML methods must demonstrate clear advantages over classical methods on real-world, post-route metrics before being considered superior.

In summary, while the RL approach (AlphaChip) represents a significant engineering effort, this assessment concludes that it does not yet surpass well-tuned classical methods or human experts in macro placement, particularly when considering the massive computational cost and instability of the training process.