💻 computer science

StableEval Arena: A Cost-Aware Agentic Benchmark for Stablecoin Price Stability Prediction

StableEval Arena is a cost-aware benchmark framework that evaluates LLM-backed agentic systems on stablecoin peg-risk prediction by measuring a joint spectrum of forecast quality, operational reliability, and computational cost, revealing that while agents reliably produce structured outputs, they struggle to accurately predict rare severe-stress and sustained-depeg events.

Original authors: Sean Wan, Dongping Liu, Luyao Zhang

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sean Wan, Dongping Liu, Luyao Zhang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive spaceship fleet, and your cargo holds are filled with "digital gold" that is supposed to always be worth exactly one dollar. This isn't just any gold; it's a special kind of money called a stablecoin, designed to stay perfectly steady so people can buy coffee, pay rent, or trade assets without worrying that their money might suddenly vanish or double in value overnight. But here's the catch: even though these coins are supposed to be boringly stable, they can sometimes panic. When the market gets scary, the price can wobble, drop below a dollar, or even crash completely. This is called "de-pegging," and if it happens, it's like your spaceship's fuel gauge suddenly lying to you.

To keep the fleet safe, we need a lookout who can spot trouble before it happens. In the past, we used simple math or human experts for this. But now, we have Agentic AI—super-smart computer programs that can think, reason, and make decisions on their own, kind of like a digital co-pilot. The big question is: Can these AI pilots actually do a better job than a simple rulebook at spotting a storm before it hits? We need to know if they are truly trustworthy, not just if they can follow instructions. If an AI says, "All clear!" right before the ship crashes, it doesn't matter how fast it is or how pretty its report looks; it's a failure.

This is exactly what the paper "StableEval Arena" sets out to test. The researchers built a giant, cost-aware training ground (an "arena") to see if these AI agents can predict when a stablecoin is about to lose its dollar-peg. They didn't just ask the AI to guess the future price; they asked it to act like a risk manager. The AI had to look at the last 30 days of price history and market chatter, then predict what would happen in the next seven days. Would the coin stay Stable? Would it enter a Watch state (a little shaky)? Or would it go into Stress (a full-blown crisis)?

The researchers ran two different kinds of tests. First, they created a "stress-test" version with 120 cases where trouble was more common, just to see if the AI could spot the danger signs when they were right in front of it. Then, they ran a "real-world" version with 507 cases that looked exactly like the natural world, where stablecoins are usually calm and only rarely have problems. They tested six different AI agents against some very simple baselines (like a rule that just says "assume everything is fine" or a basic math model).

The results were a mix of good news and a serious reality check. On the good side, the AI agents were incredibly reliable at following the rules. They produced perfectly formatted reports, didn't crash, and did it all for a very low cost. They were great at saying, "Hey, things look a little wobbly today." However, when it came to the most important job—spotting a rare, catastrophic crash—the AI agents mostly missed the mark. In the big test of 507 cases, the AI missed 19 out of 21 actual stress events. They were so good at following the protocol that they produced valid reports, but those reports often said "All clear" right before the coin actually crashed.

In fact, in the real-world test, even a simple computer program that just looked at past trends struggled to predict the rare crashes, missing the majority of them just like the AI. The fancy AI agents did not dominate or outperform these classical methods; in fact, none of the agents could reliably detect the rare, high-stakes stress events that mattered most. The paper concludes that while these AI agents are excellent at being obedient and efficient, they aren't yet trustworthy enough to be the sole guardians of our digital money. They can follow the script perfectly, but they still struggle to see the rare, scary storms that matter most. The researchers suggest that until AI can get better at spotting these rare, high-stakes events, we shouldn't let them drive the spaceship alone.

Technical Summary: StableEval Arena

Problem Statement

Stablecoins are critical infrastructure for decentralized finance, relying on a fragile promise of maintaining a one-dollar peg. While existing cryptocurrency forecasting research focuses on predicting full price paths using multimodal data, stablecoin risk assessment requires a different objective: determining whether a stablecoin will remain stable, enter a warning state, or experience peg stress over a fixed future window.

Current evaluation frameworks for agentic AI systems often prioritize final-answer accuracy, neglecting operational reliability, protocol compliance, and computational cost. Furthermore, in stablecoin domains, high aggregate accuracy can be misleading due to the overwhelming prevalence of "stable" cases, potentially masking a system's inability to detect rare but catastrophic depegging events. There is a lack of benchmarks that evaluate agentic systems on stablecoin risk assessment under strict cost constraints, leakage-safe conditions, and protocols that distinguish between procedural reliability (following instructions) and financial-risk reliability (detecting rare stress).

Methodology

Benchmark Design: Two-Block Architecture

StableEval Arena introduces a historical-replay benchmark designed to evaluate agentic AI systems on predicting stablecoin peg stability. The benchmark utilizes a "leakage-safe" design where agents receive a frozen 30-day lookback window of hourly data and must predict the risk status for the subsequent 7-day (168-hour) hidden horizon.

The evaluation is conducted across two complementary experiment blocks:

StableEval-120-Enriched: A stress-enriched validation block containing 120 cases (54 stable, 45 watch, 21 stress). This block concentrates rare "watch" and "stress" outcomes to provide sharper diagnostic coverage for rare-risk detection.
StableEval-507-Natural: A full-pool scaling block containing 507 cases (441 stable, 45 watch, 21 stress) drawn from a natural distribution. This block tests system robustness and behavior under realistic class imbalances where stable cases dominate.

Task Formulation

For each case, the system receives:

Target Assets: USDT, USDC, and DAI.
Context: BTC and ETH market data.
Features: Derived from 720 hours of hourly OHLCV data, including peg deviation, volatility, volume signals, and market context.
Constraint: Agents must use only the provided lookback window; future prices and labels are strictly hidden.

Output Requirement: Agents must produce a structured JSON object containing:

Stability class label: stable, watch, or stress.
Depeg probability ($0-1$).
Maximum peg deviation estimate (in basis points).
Deviation direction ( $-1, 0, +1$ ).
Confidence score ($0-1$).
Evidence-based rationale.

Agent Systems and Baselines

The benchmark evaluates six LLM-backed agentic systems against non-agent baselines:

Native/Framework Agents: Nanobot, LangGraph, Hermes AI, EvoAgentX.
Adapter-Based Agents: OpenClaw, NanoClaw (executed via StableEval adapters).
Baselines: Always-stable, historical persistence, threshold rules, and simple machine learning (logistic regression).

All systems run on a shared backend (OpenRouter with qwen/qwen-2.5-72b-instruct) using a frozen protocol, identical prompt contracts, and strict JSON parsing.

Evaluation Metrics

The framework moves beyond simple accuracy to measure:

Prediction Quality: Macro-F1, Balanced Accuracy, Brier score (for probability calibration), and Mean Absolute Error (for deviation magnitude).
Risk Reliability: Stress Recall (ability to detect stress cases), Missed-Depeg Rate (MDR), and False Alarm Rate (FAR).
Operational Metrics: Valid JSON output rate, latency, token consumption, and estimated inference cost.

Key Results

1. Procedural vs. Financial Reliability Gap

The most significant finding is the divergence between procedural reliability and financial-risk detection.

Procedural Reliability: All evaluated agentic systems achieved a 100% valid JSON output rate across both the 120-case and 507-case blocks, demonstrating high adherence to the output schema and protocol.
Financial Reliability: Despite valid outputs, agents failed to reliably detect rare stress events. In the 507-case pool, all six agentic systems missed 19 out of 21 true stress cases. No agentic system achieved a missed-depeg rate below 1.0000 for sustained depegs.

2. Performance Across Blocks

120-Case (Stress-Enriched): Calibrated agentic systems (e.g., Nanobot) showed improved aggregate Macro-F1 (0.351) compared to simple baselines (0.285 for historical persistence). However, stress recall remained low (0.095 for the best agent vs. 0.333 for historical persistence), indicating that while agents improved general classification, they still struggled with the most severe, rare events.
507-Case (Natural-Distribution): Under natural class balance, simple baselines outperformed agentic systems. The "Always-Stable" baseline achieved 0.870 accuracy but zero stress recall. The simple ML/logistic baseline achieved the highest Macro-F1 (0.402) and balanced accuracy (0.493), outperforming all agentic systems. This highlights that in stable-dominant environments, simple rules often suffice for aggregate metrics, while agents fail to add value in detecting rare risks.

3. Cost and Calibration

Calibration: Applying a calibration rule to map continuous risk estimates to discrete labels improved Macro-F1 in the 120-case block (from 0.301 to 0.351) but did not resolve the fundamental issue of missing rare stress events.
Cost: Agentic systems incurred non-zero inference costs (ranging from ~$0.0029 to $0.0101 per case), whereas deterministic baselines had zero cost. A verifier layer improved stress recall slightly (to 0.143) but at a significant cost penalty (tripling the cost for Nanobot) with diminishing returns on overall Macro-F1.

Significance and Claims

The paper claims that StableEval Arena is not a demonstration that current agentic systems are ready for deployment as stablecoin risk monitors. Instead, its significance lies in exposing a critical gap: agentic systems can reliably follow protocols and produce valid structured outputs, yet they remain unreliable for detecting rare, high-consequence financial stress events.

Key contributions include:

A Cost-Aware, Leakage-Safe Benchmark: A framework that evaluates agents on prediction quality, operational cost, and reliability simultaneously, using historical replay to prevent data leakage.
Two-Block Diagnostic Design: The separation of a stress-enriched block (for diagnosing rare-risk sensitivity) and a natural-distribution block (for testing robustness under realistic imbalances) prevents misleading conclusions that might arise from aggregate leaderboards alone.
Redefining Trustworthiness: The work argues that in financial monitoring, "trustworthiness" is a joint property of forecast quality, operational reliability, and computational cost. A system that produces valid JSON but misses depegs is operationally reliable but financially untrustworthy.
Open Resources: The authors release the dataset on Hugging Face and source code on GitHub to support auditing, replication, and future research into trustworthy agentic AI infrastructure.

The paper concludes that while agentic AI shows promise in processing bounded evidence and generating structured assessments, the detection of rare peg stress and sustained depegs remains a significant limitation. Future work must focus on richer evidence packets (on-chain data, oracle feeds, governance signals) and risk-sensitive cost functions that explicitly penalize missed stress events.