CryptoAnalystBench: Failures in Multi-Tool Long-Form LLM Analysis

1. Problem Statement

The paper addresses a critical gap in the evaluation of Large Language Model (LLM) agents: the inability of current benchmarks to assess long-form, multi-tool reasoning in high-stakes, data-dense environments.

• The Gap: Existing benchmarks focus on either short-answer tool calling (e.g., "fetch the price of X") or static knowledge retrieval. They fail to capture the complexity of modern analyst agents that must orchestrate dozens of tool calls, synthesize heterogeneous data (structured APIs, unstructured documents, code), and produce long-form analytical reports.
• The Challenge: In domains like cryptocurrency and DeFi, data is volatile (changing hourly), high-volume, and often contradictory across sources. Agents must not only retrieve data but also reconcile conflicts, maintain temporal awareness, and synthesize actionable insights without hallucinating or misattributing sources.
• The Risk: Failures in these systems lead to poorly grounded financial decisions, misestimated risks (e.g., incorrect Total Value Locked figures), and flawed investment allocations.

2. Methodology

The authors propose a comprehensive framework consisting of a new benchmark, an agentic evaluation harness, and a multi-layered evaluation pipeline.

A. CryptoAnalystBench (The Benchmark)

• Source: Derived from real production user traffic (198 curated queries) rather than synthetic data.
• Scope: Covers 11 categories of crypto/DeFi analysis, including project research, market data, on-chain analytics, and macro narratives.
• Design Principles:
  • Production Realism: Queries reflect actual user intent and language.
  • Time-Robustness: Questions remain valid over time, though answers depend on current data.
  • Complexity: Requires web search, tool calling, and multi-step synthesis, not just factual recall.

B. Agentic Harness

• Architecture: A ReAct-style loop where an LLM agent processes a query, performs semantic rewriting, and iteratively invokes tools (CoinGecko, DefiLlama, blockchain explorers, search APIs).
• Output: Generates long-form, structured responses mimicking professional analyst reports.
• Models Tested: Evaluated across frontier models including GPT-5.2, Kimi K2.5, Qwen 235B, GLM 4.7, and GPT OSS 120B.

C. Evaluation Pipeline

The authors employ a three-pronged evaluation strategy:

1. LLM-as-a-Judge (Rubric Scoring): Scores responses on four dimensions (1–10 scale):
   • Temporal Relevance: Freshness of data.
   • Data Consistency: Internal coherence and lack of contradictions.
   • Depth: Technical detail and multi-axis coverage.
   • Relevance: Directness to user intent.
2. Automated Citation Verification: A pipeline that extracts atomic claims, maps them to tool outputs, and classifies them as Exact, Derived, or Fabricated. It also measures citation preciseness.
3. Human Expert Annotation: Domain experts annotated a subset of outputs to validate the LLM judge and identify qualitative failure modes.

3. Key Contributions

1. CryptoAnalystBench: The first production-aligned benchmark for long-form, multi-tool crypto analysis, featuring 198 queries and an associated evaluation harness.
2. Failure Mode Taxonomy: Through human annotation, the authors identified seven higher-order error types that scalar scores miss:
   • Staleness/Missing Time Bounds: Outdated info or unclear validity windows.
   • Inconsistent Claims: Internal contradictions.
   • Source Reconciliation Failure: Inability to resolve conflicting data from multiple tools.
   • Shallow Synthesis: Aggregating facts without meaningful integration.
   • Missing Risk/Mechanism Context: Omitting necessary assumptions or risk factors.
   • Overconfident Prediction: Strong claims without uncertainty calibration.
   • Partial/Misframed Answer: Addressing only part of the query.
3. Refined Evaluation Metrics: Demonstrated that while LLM judges align reasonably well with humans on relevance and temporal scores (Kappa ~0.35–0.42), they struggle with precise scalar calibration. However, they are highly effective at identifying specific failure categories (93.45% accuracy in automated error classification).
4. Mitigation Strategies: Proposed and tested three interventions:
   • Prioritizing structured API outputs over unstructured search results.
   • Dynamically augmenting prompts with temporal context.
   • Activating task-specific sub-prompts for depth.
   • Finding: Mitigations improved performance for strong models but caused regressions in weaker models, suggesting interventions must be model-aware.

4. Key Results

• Model Performance:
  • Kimi K2.5 emerged as the strongest generalist, achieving high depth without sacrificing relevance.
  • GPT-5.2 excelled in data consistency but showed more task-dependent behavior.
  • GLM 4.7 and Qwen 235B showed low hallucination rates (<2% fabricated claims).
• Hallucination vs. Structural Failure:
  • Fabricated claims were low (<6% across all models).
  • Primary Failure Mode: The dominant errors were structural, not factual. Models struggled with Source Reconciliation (handling conflicting data) and Temporal Grounding (using outdated data despite having access to fresh tools).
• Citation Quality:
  • Citation preciseness was high (>85% for most models), indicating models generally know where they got data, but often fail to reconcile conflicting sources or provide the necessary context for that data.
• Evaluation Alignment:
  • LLM judges correlated moderately with humans on rubric scores but were highly reliable in detecting the presence of specific error types (e.g., "Staleness" or "Overconfidence").

5. Significance and Future Directions

• Paradigm Shift: The paper argues that evaluating analyst agents requires moving beyond "fact-checking" to assessing synthesis quality, conflict resolution, and contextual framing.
• Scalable Feedback: While human annotation is expensive, the study validates that LLM judges can serve as a scalable feedback mechanism for developers, provided they are used to detect specific failure modes rather than just assigning scalar scores.
• Generalizability: While focused on crypto, the findings apply to any high-stakes domain requiring multi-tool reasoning (legal, medical, enterprise intelligence), where data volatility and source reconciliation are critical.
• Open Challenges: The authors highlight the need for adaptive mitigation strategies (tailoring interventions to model capability) and better methods for evaluating temporal awareness in dynamic environments.

In conclusion, CryptoAnalystBench provides the first systematic lens into how advanced LLM agents fail when tasked with complex, real-world analysis, revealing that the bottleneck is no longer simple fact retrieval, but the synthesis and reconciliation of volatile, multi-source data.