💬 NLP

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

Relay-Bench is a holistic, text-only benchmark designed to evaluate large language models on multi-domain reasoning chains by presenting unsaturated composite problems that require integrating diverse skills like coding, math, and web search within a single prompt, where the leading model GPT-5.5 (xHigh) achieves a score of 43.3%.

Original authors: Liam Swayne

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liam Swayne

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your favorite smart assistant isn't just a trivia bot or a code-cruncher, but a true multitasking wizard. In the rapidly evolving field of Artificial Intelligence, researchers are constantly building "benchmarks"—which are essentially standardized tests—to see how smart these digital brains really are. For a long time, these tests were like separate subjects in school: one exam for math, another for history, and a third for coding. But real life doesn't work that way. When you ask a human to plan a trip, they don't just calculate distances; they check the weather, read reviews, book a hotel, and maybe even write a funny email to a friend, all in one go. This paper dives into a specific corner of AI research called "reasoning chains," where the goal is to see if a model can juggle many different types of tasks at once without dropping the ball. The big question everyone cares about is: Can these AI models handle the messy, complicated, multi-step problems of the real world, or do they get confused and give up when things get too crowded?

Enter Relay-Bench, a new, super-challenging test designed to see if AI models can run a "relay race" of the mind. Instead of asking a model to solve one math problem or write one poem, the creators of this test string together a whole chain of different tasks—like solving a math puzzle, finding a specific fact on the web, writing a bit of code, and decoding a secret message—all inside a single, massive prompt. Think of it like a video game level where you have to jump over a pit, solve a riddle to open a door, and then fight a boss, all without hitting the "reset" button. The researchers built 31 of these complex challenges, some of which are so long and cluttered with fake information that they feel like reading a novel just to find a single number. They also added a "secret code" layer, where every word in the instructions is replaced with a strange three-letter code, forcing the AI to decode the message before it can even start solving the problems.

When they ran the top three AI models of the time (GPT-5.5, Gemini 3.1 Pro, and Claude Opus 4.7) through this gauntlet, the results were a bit of a shock. Even the smartest model, GPT-5.5, only got about 43.3% of the answers right. That's less than half! The other models scored even lower, with one of them only managing 16.7%. The paper suggests that while these AI models are getting very good at single tasks, they struggle significantly when asked to chain multiple different skills together, especially when the instructions are confusing or the task is very long. In fact, the test was so hard that many models simply gave up or got stuck in loops, unable to finish the race.

The researchers also found that these models sometimes "hallucinate," which is a fancy way of saying they make up answers when they don't know the solution. One model, Claude Opus 4.7, was very cautious; it refused to answer many of the encoded questions entirely, likely because its safety filters were too strict. Another model, Gemini 3.1, tried to guess more often but ended up with more wrong answers. The study highlights that current AI models are still far from being the all-around experts we hope they will be. They are like brilliant students who can ace a math test but might panic if you ask them to do math while simultaneously translating a poem and searching for a recipe. The authors suggest that it might take another year or two before AI models get good enough to consistently pass this kind of "relay race" test, and they warn that as models get smarter, these tests will need to get even harder to keep up.

Technical Summary: Relay-Bench

Problem Formulation

The paper addresses a critical gap in the evaluation of Large Language Models (LLMs): the lack of an unsaturated, holistic, text-only benchmark that measures the ability to complete tasks from distinct domains within a single prompt. While frontier models have achieved saturation (Pass@1 > 90%) on many single-domain benchmarks (e.g., GPQA, MATH-500) and specialized agentic benchmarks, they struggle with "multi-domain reasoning chains." The authors posit that the ability to reason across multiple domains in combination is a "critical impediment in the pursuit of artificial general intelligence."

Existing benchmarks often suffer from saturation, rely on non-text inputs (e.g., MMMU), or use complex scoring frameworks that obscure model performance. Relay-Bench is designed to be an orthogonal approach that increases difficulty by layering complexity within the problems themselves rather than through input modalities or scoring systems.

Methodology

Benchmark Structure

Relay-Bench consists of 31 problems, 30 of which form a held-out private test set to prevent contamination. Each problem is a composite problem composed of 2 to 13 subproblems.

Subproblems: These are single-domain tasks (e.g., visual reasoning, coding, math, information extraction, general knowledge, data analysis) similar to standard QA benchmarks.
Dependencies: Subproblems are strung together with dependencies; the final answer depends on the outputs of previous subproblems. The dependency graph is acyclic.
Augmentation: To increase difficulty and simulate real-world "context bloat," problems undergo two specific augmentations:
1. Context Bloat: Irrelevant "user preferences" (approx. 7,813 characters) are inserted into the prompt.
2. Prompt Encoding: Words, symbols, and numbers are substituted with three-letter alphabetic strings (e.g., "bottle" $\to$ ""). Models are provided a dictionary and must decode the prompt before solving. Some problems require over 10,000 translation operations.

Evaluation Protocol

Text-In, Text-Out: The benchmark avoids visual inputs by converting visual reasoning tasks (e.g., mazes) into text-based representations (e.g., ASCII art).
Tool Permissiveness: Unlike many benchmarks that disable tools, Relay-Bench explicitly encourages models to use all available capabilities, including code execution and web search.
Scoring: The benchmark uses a Pass@1 metric. Grading is deterministic and exact; models are instructed to output specific string formats (capitalization, symbols) to allow for script-based verification without LLM judges or "quasi-exact" matching.
Models Evaluated: The paper evaluates three frontier models with maximum "thinking effort" settings:
- Claude Opus 4.7 (Max)
- Gemini 3.1 Pro (High)
- GPT-5.5 (xHigh)

Key Contributions

Relay-Bench Benchmark: The introduction of a holistic, text-only benchmark designed to remain unsaturated by combining multi-domain reasoning with prompt encoding and context bloat.
Novel Difficulty Mechanisms: The use of prompt encoding (requiring massive translation operations) and deliberate context bloat to "exhaust" models and test instruction-following reliability under noise.
Tool-Permissive Evaluation: A framework that reflects user experiences by allowing full tool access (search, code), contrasting with benchmarks that restrict these features.
Exactness in Grading: A shift from probabilistic or LLM-judged scoring to deterministic, exact string matching to ensure reproducibility.

Results

Performance

Leading Model: GPT-5.5 (xHigh) achieved the highest score of 43.3%.
Comparative Scores:
- Gemini 3.1 Pro (High): 40.0%
- Claude Opus 4.7 (Max): 16.7%
Benchmark Comparison: All models scored significantly lower on Relay-Bench than on other benchmarks. For instance, the average Relay-Bench score was 33.3%, compared to 42.9% on Humanity's Last Exam (HLE), 79.3% on ARC-AGI-2, and 92.7% on $\tau^2$ -Bench.
Saturation: The paper suggests Relay-Bench may take 1–2 years to reach saturation, whereas other benchmarks often saturate within two years.

Error Analysis & Hallucination

Parseability: Only 50 of 90 total responses contained a parseable final answer.
Hallucination Rates:
- Claude Opus 4.7 had the lowest hallucination rate (only 1 incorrect parseable answer) but also the fewest parseable answers overall.
- GPT-5.5 and Gemini 3.1 Pro had higher hallucination rates.
- The paper notes that the Pass@1 scoring framework (where non-response equals an incorrect answer) incentivizes guessing over abstaining, yet models still exhibited vastly different hallucination behaviors.
Failure Modes:
- Models frequently failed to solve specific subproblems and then guessed the final composite answer rather than stopping.
- Claude Opus 4.7 explicitly halted responses (refusal) on all encoded problems in the subset, suggesting overly conservative safety safeguards.
- Problems exceeding 15,000 characters were significantly harder; only GPT-5.5 and Gemini 3.1 Pro solved one of seven such problems.

Cost Analysis

Total Development Cost: $163.29 (including test runs and question generation).
Model Costs:
- GPT-5.5 was the cheapest to evaluate ($22.44 total), despite high token usage, due to lower per-token pricing.
- Claude Opus 4.7 was the most expensive ($33.76), driven by high input token usage (over 2 million tokens on six problems) due to extensive chains of thought and tool iterations.
- Gemini 3.1 Pro had the lowest token prices but suffered from a very low input cache hit rate (0.6%), resulting in higher costs than GPT-5.5.
Efficiency: GPT-5.5 Pareto-dominates the other models, offering the best balance of accuracy and cost.

Significance and Claims

The paper claims that Relay-Bench provides a more rigorous and realistic assessment of LLM capabilities for general users by:

Testing Reliability: Layering generalist tasks (instruction following, string manipulation) on top of expert-level challenges to measure reliability under complex, multi-step workflows.
Avoiding Saturation: By combining domains and adding encoding complexity, the benchmark avoids the rapid saturation seen in single-domain QA tasks.
Reflecting Real-World Usage: By enabling tools and simulating "noisy" prompts (context bloat), it mirrors the conditions users face when deploying agentic workflows.

The authors modestly note that the small test set size (30 problems) results in wide confidence intervals for model performance. They acknowledge that the benchmark's reliance on web search and code execution introduces maintenance challenges (e.g., link rot) and that future iterations might narrow the scope to better isolate reasoning capabilities from tool usage. The paper concludes that while current frontier models show promise, their ability to reliably execute long, multi-domain reasoning chains remains a significant challenge.