RISCBench: Benchmarking RISC-V Orchestration Efficiency in FPGA and FPGA-Like Computing Engines

This paper introduces RISCBench, a benchmark suite and methodology that quantifies orchestration efficiency in heterogeneous RISC-V systems using a new Sustained Instantaneous Throughput (SIT) metric to address the limitations of conventional performance indicators in FPGA and accelerator-class platforms.

The Gist

Imagine you are running a massive, high-tech kitchen. You have incredible, super-fast ovens (the accelerators) that can cook thousands of dishes in a second. But, you also have a head chef (the control core) whose job isn't to cook, but to tell the ovens when to start, move ingredients around, and make sure the dishes don't collide.

For years, when people evaluated these kitchens, they only looked at the ovens. They asked, "How many dishes can this oven cook in one second?" They measured the peak speed (like FLOPs or TOPS).

The Problem:
The authors of this paper, Dave, Preethi, and their team, realized that measuring the oven's speed is useless if the chef is slow, confused, or running out of ingredients. Even if your oven can cook 1,000 pizzas a minute, if your chef takes 10 minutes to decide where to put the cheese, your actual output is terrible.

In the world of computers (specifically FPGAs and RISC-V chips), the "chef" is the control system. As computers get more complex, the "chef" often becomes the bottleneck, slowing everything down.

The Solution: RISCBench
The team created a new tool called RISCBench. Think of this as a new way to judge a kitchen. Instead of just timing how fast the oven can go, they time how fast the kitchen actually works over a whole dinner service.

They introduced a new metric called SIT (Sustained Instantaneous Throughput). Here is the best way to understand it:

• The Old Way (Peak Speed): Imagine a race car driver hitting 200 mph for exactly 3 seconds on a straightaway. The old metrics say, "Wow, 200 mph! That's a fast car!"
• The New Way (SIT): The SIT metric looks at the whole race. It sees that the driver hit 200 mph for 3 seconds, but then spent 2 minutes stuck in traffic, waiting for a pit crew, and getting lost. The SIT score reflects the average speed over the whole race, not just the 3 seconds of glory.

What They Found:
They tested this on two types of computer setups:

1. The Soft Core: Like a chef working in a temporary, makeshift kitchen (a standard FPGA).
2. The Hard Core: Like a chef in a state-of-the-art, permanent kitchen with dedicated staff (an advanced accelerator chip).

They ran a test where the "chef" had to move ingredients (data) around constantly.

• At the start: The kitchen ran perfectly. The ovens were blazing hot, and the output was huge.
• A moment later: The "chef" got overwhelmed. The instructions to move ingredients got backed up. The ovens had to wait. The speed dropped.

The old metrics would have ignored this drop because they only looked at the first few seconds. RISCBench and SIT caught this drop. They showed that as computers get more tightly packed together, the "chef" (orchestration) becomes the most important part of the system. If the chef isn't efficient, the super-fast ovens are wasted.

Why It Matters:
This is a big deal for AI and future computing. As we build smarter chips for things like self-driving cars or AI assistants, we can't just make the "engines" faster. We have to make sure the "control system" is just as good at managing the chaos.

The Takeaway:
The authors have made their tool open-source (free for everyone to use). They are inviting other engineers and researchers to stop just measuring the "top speed" of computers and start measuring how well they perform over time. It's a shift from asking "How fast can it go?" to "How fast does it go when things get real?"

Technical Summary

Based on the provided paper "RISCBench: Benchmarking RISC-V Orchestration Efficiency in FPGA and FPGA-Like Computing Engines," here is a detailed technical summary:

1. Problem Statement

As heterogeneous computing systems evolve toward tighter integration of accelerators and reconfigurable fabrics, they increasingly rely on lightweight control-plane cores (often RISC-V) to manage data movement, synchronization, and scheduling. However, current evaluation metrics are insufficient for assessing these systems:

• Limitations of Conventional Metrics: Standard descriptors like FLOPs, TOPS/W, and energy per operation focus primarily on arithmetic capability and peak performance.
• The Orchestration Blind Spot: These metrics fail to capture orchestration efficiency. In tightly coupled systems, the behavior of the control plane (scheduling, coordination, data placement) often becomes the primary bottleneck, determining the sustained throughput rather than the theoretical peak.
• The Gap: There is a lack of methodology to quantify how orchestration overheads degrade performance over time, leading to an overestimation of system capabilities in real-world scenarios.

2. Methodology

To address this gap, the authors introduce RISCBench, a lightweight benchmark suite and open methodology designed specifically to quantify orchestration efficiency.

• Core Metric: Sustained Instantaneous Throughput (SIT):
  • Unlike peak-oriented descriptors, SIT accumulates the contribution of high-efficiency execution windows to calculate realized sustained throughput.
  • It specifically measures how coordination, scheduling, and data placement affect the persistence of high-efficiency execution under realistic orchestration conditions.
• Experimental Setup:
  • Platforms: The benchmark was prototyped on a Nios-V soft RISC-V core on an FPGA (to validate integration in reconfigurable environments) and evaluated on an accelerator-class platform embedding numerous hard RISC-V control cores.
  • Workload: SRAM-resident tiled matrix kernels were utilized. This design minimizes external memory effects to isolate and expose pure orchestration behavior.
  • Analysis: Execution traces were analyzed to observe throughput degradation patterns over time.

3. Key Contributions

• RISCBench Suite: An open-source benchmark suite (available at www.riscbench.com) that enables the chip architecture, EDA research, and systems communities to evaluate orchestration efficiency.
• The SIT Metric: A novel, platform-independent descriptor that complements traditional peak-based metrics by focusing on sustained execution behavior.
• Identification of Orchestration Bottlenecks: The methodology explicitly exposes inefficiencies caused by synchronization overhead and memory residency transitions that are invisible to standard benchmarks.

4. Results and Observations

Experimental execution traces (referenced as Fig. 1 in the paper) revealed distinct performance phases:

• Initial Phase: Early execution windows demonstrate near-aggregate throughput, indicating that the system can achieve high efficiency when data is fully resident on-chip and coordination overhead is low.
• Degradation Phase: As execution continues, performance declines due to the accumulation of synchronization overhead and memory residency transitions.
• Peak vs. Sustained: While peak throughput is observed under ideal, full on-chip residency conditions, sustained performance drops significantly as coordination complexity and bandwidth contention increase.
• Conclusion: As heterogeneous integration advances, orchestration behavior increasingly dictates realized throughput, often becoming the limiting factor before arithmetic limits are reached.

5. Significance

• Paradigm Shift in Evaluation: RISCBench moves the industry focus from "peak potential" to "realized sustained performance," providing a more accurate picture of system utility in AI inference and other data-intensive applications.
• Design Guidance: By highlighting orchestration-induced degradation, the benchmark guides architects to optimize control-plane logic, scheduling algorithms, and data placement strategies rather than just raw compute density.
• Open Ecosystem: By releasing the suite as open-source, the authors foster reproducibility and encourage broader adoption across the FPGA and accelerator design communities, ensuring that future hardware developments address control-plane bottlenecks directly.

In summary, RISCBench provides the necessary tools to measure the "glue" that holds heterogeneous systems together, arguing that without efficient orchestration, high peak arithmetic performance cannot be sustained in real-world workloads.