💬 NLP

Benchmarking Real-Time Question Answering via Executable Code Workflows

This paper introduces RT-QA, a dynamic evaluation framework using executable code workflows to assess real-time question answering, revealing that even state-of-the-art models struggle with temporal adaptability due to lazy retrieval and temporal confusion, achieving only 46% accuracy.

Original authors: Wenjie Zhou, Yuan Gao, Xin Zhou, Hao Fu, Zhongjian Miao, Wei Chen, Bo Chen, Xiaobing Zhao

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenjie Zhou, Yuan Gao, Xin Zhou, Hao Fu, Zhongjian Miao, Wei Chen, Bo Chen, Xiaobing Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🕵️‍♂️ The Big Problem: The "Yesterday's Newspaper" Trap

Imagine you ask a super-smart librarian (an AI) a question: "What was the headline in the newspaper yesterday?"

Most current AI models are like librarians who have memorized every book in the library up until a certain date, but they haven't stepped outside since then. If you ask them about yesterday, they might:

Guess based on old patterns (Hallucination).
Recall a headline from last year because they think "yesterday" is a fixed memory (Knowledge Obsolescence).
Refuse to answer because they don't have a "live" connection to the newsstand.

Existing tests for these AIs are like pop quizzes using old, static questions. They don't tell us if the AI can actually go out, check the current newsstand, and bring back the real answer for today.

🚀 The Solution: RT-QA (The "Live Agent" Test)

The authors created a new test called RT-QA. Instead of giving the AI a static question with a pre-written answer key, they gave it a mission.

The Analogy: The "Magic Recipe" vs. The "Pre-cooked Meal"

Old Way (Static QA): The test gives the AI a pre-cooked meal (the answer) and asks, "Is this tasty?" The AI just has to say "Yes" or "No." It doesn't need to cook anything.
New Way (RT-QA): The test gives the AI a Magic Recipe (executable code). The recipe says: "Go to the National Stadium website, look at the schedule for tomorrow, and tell me who is playing."
- The AI has to run the recipe (execute the code).
- It visits the website right now.
- It reads the schedule right now.
- It generates the answer based on the current reality.

If the website changes its layout (like a restaurant changing its menu font), the AI's "Magic Recipe" might break. The RT-QA system includes a Self-Repair Mechanism—like a robot chef that notices the oven is broken, fixes the wiring, and keeps cooking.

📊 What They Found: The "Lazy Student" and the "Time-Traveler"

The researchers tested top-tier AI models (like GPT-5.2, Claude, and others) on this new test. The results were surprising and a bit embarrassing for the AIs.

1. The Score was Low
Even the smartest models only got about 46% correct. That's barely passing a high school exam. This means current AIs are terrible at handling "live" information.

2. Failure Mode #1: The "Lazy Student" (Lazy Retrieval)

The Scenario: The question asks for the stock price of a company.
The Mistake: The AI looks at the search engine's summary (the "snippet" at the top of Google) and guesses the answer without actually clicking the link to the stock exchange.
The Metaphor: It's like a student who reads the table of contents of a textbook and thinks they know the whole story, without ever reading the actual chapters. They rely on shortcuts and miss the details.

3. Failure Mode #2: The "Time-Traveler" (Temporal Confusion)

The Scenario: The question asks, "Who won the concert at the National Stadium yesterday?"
The Mistake: The AI searches for "National Stadium concert" and finds a big article about a concert that happened in 2024. It gets excited, grabs that date, and forgets to check what day it is today. It answers with the 2024 winner.
The Metaphor: Imagine a time traveler who lands in 2026 but keeps looking at their watch from 2024. They get confused about what "today" means.
The Twist: The researchers found that harder questions (Level 3) were sometimes answered better than medium ones (Level 2). Why? Because the hard questions forced the AI to think so hard and plan so carefully that it accidentally fixed its time confusion. The medium questions were "tricky" enough to fool the AI but not hard enough to force it to slow down and double-check the date.

🛠️ How They Built It (The "Agent-in-the-Loop")

To make this test, they didn't write 320 questions by hand. They built a Robot Builder (the RT Agent).

The Builder looks at a website (like a weather site or a sports schedule).
It writes a Python script (a small computer program) to scrape that data.
It tests the script. If the script breaks because the website changed, the Builder fixes the code automatically.
This allows them to create a massive, ever-updating test bank that covers 12 different topics (Finance, Sports, Weather, etc.).

💡 The Takeaway: What's Next?

The paper concludes that simply giving AI better search tools isn't enough. We need to teach them Time Management.

Current AI: "I found a fact about 2024. That must be the answer."
Future AI: "I found a fact about 2024. Wait, the user asked about yesterday. I need to discard that old fact, find the current date, and look for the new fact."

In short: The internet is a living, breathing thing that changes every second. Our AI assistants are currently like tourists with a map from 1990. RT-QA is the first test that forces them to learn how to use a GPS and navigate the real world, right now.

1. Problem Statement

Current Large Language Model (LLM) agents are increasingly integrated with search capabilities to access real-time information. However, existing evaluation benchmarks suffer from two critical limitations:

Static Nature: Most benchmarks rely on fixed question-answer pairs. This leads to Data Leakage (models answering from training memory rather than searching) and Knowledge Obsolescence (facts changing over time render ground truth invalid).
Dependency on Structured Sources: Recent attempts to create dynamic benchmarks often rely on pre-existing APIs or structured databases (e.g., Wikidata). These fail to cover high-frequency, unstructured information from the open web (e.g., daily stock prices, specific event schedules) where no direct API exists.

The core challenge is evaluating an agent's ability to handle relative temporal constraints (e.g., "yesterday," "past 7 days") and retrieve up-to-date answers from unstructured web sources without relying on pre-stored data.

2. Methodology: The RT-QA Framework

The authors propose RT-QA, a dynamic evaluation framework that shifts the paradigm from "Static Answers" to "Executable Code Workflows."

A. Core Innovation: Executable Workflows

Instead of storing a static answer string, each test item in RT-QA is a tuple $(q_\tau, w_\tau)$ :

$q_\tau$ : A natural language question with relative time constraints.
$w_\tau$ : A Python-based executable workflow.
Ground Truth Generation: The ground truth is not pre-calculated. It is generated dynamically at the exact moment of evaluation by executing $w_\tau$ . The workflow visits authoritative websites, parses raw DOM elements, and computes the answer based on the current timestamp.

B. The RT Agent Pipeline

To construct this benchmark at scale, the authors developed an Agent-in-the-Loop pipeline:

Preparation: Curating 12 domains (Finance, Sports, Policy, etc.) and defining tools (e.g., Check_html_content, Code_interpreter).
Workflow Generation: An LLM (RT Agent) autonomously analyzes a target URL, plans a strategy, and generates Python code (using Playwright) to scrape and extract data.
Self-Repair Mechanism: Since web structures change, the pipeline includes a Repair Agent. If a workflow fails (e.g., CSS selector broken), the agent re-analyzes the live DOM, identifies structural changes, and patches the code to restore functionality.
Verification: The agent cross-validates the extracted text against page screenshots to ensure alignment with human perception.

C. Dataset Construction

Scale: 320 questions across 12 domains.
Difficulty Levels:
- L1 (Direct): Direct retrieval (e.g., "What is the weather in Beijing?").
- L2 (1-Hop): Requires identifying an entity from a past event before retrieving current data (e.g., "Who hosted the 2025 Summit? What is the weather there today?").
- L3 (2-Hop+): Complex multi-step reasoning involving multiple entities and timeframes.

3. Key Contributions

RT-QA Benchmark: The first benchmark to use executable code workflows to generate dynamic ground truth, eliminating data leakage and knowledge obsolescence.
RT Agent Framework: An automated system for generating and maintaining these workflows, featuring a self-repair mechanism to handle web volatility.
Failure Taxonomy: A fine-grained analysis of over 7,000 agent trajectories revealing specific failure modes in real-time reasoning.

4. Experimental Results

The authors evaluated 8 state-of-the-art models (including GPT-5.2, GLM-4.7, Claude-4.5, and DeepSeek-V3.2) over a 6-day period in January 2026.

Low Overall Accuracy: Even the best models achieved only ~46% accuracy. Most open-weight models fell below 40%.
Volatility: Performance fluctuated significantly day-to-day (e.g., DeepSeek-V3.2 varied by >11 points in 24 hours), indicating instability in retrieval strategies.
The "U-Shaped" Performance Curve: Contrary to expectations, L2 (Intermediate) tasks performed worse than L3 (Complex) tasks.
- L2 Failure: Agents often get "stuck" on historical dates retrieved during the first hop (e.g., the date of a past event) and fail to re-anchor to the current time for the second hop.
- L3 Success: The higher complexity of L3 tasks forces agents to engage in more rigorous planning, inadvertently helping them distinguish between historical and current time variables.

5. Failure Mode Analysis

Analysis of 7,000+ trajectories identified seven failure modes, with the top two being:

Lazy Retrieval (20%): Agents rely on search engine snippets rather than visiting the actual authoritative website to verify details.
Temporal Confusion (20%): A cognitive error where agents retrieve a historical date (e.g., "Oct 2025") and mistakenly use it as the "current time" for subsequent reasoning, failing to update the temporal state.
Other Modes: Incomplete scanning (15%), Hallucination (15%), Source Quality issues (10%), Parsing failures (10%), and Logic errors (10%).

6. Significance and Conclusion

The paper highlights that current agents are fragile in dynamic environments. The primary bottleneck is not just retrieval capability but Temporal State Management.

Implication: Simply providing better search tools is insufficient. Future agents require robust mechanisms to distinguish between past events and current goals, and to actively manage temporal context throughout multi-step reasoning.
Future Direction: Agents must evolve from passive searchers into proactive investigators capable of verifying information and managing temporal variables dynamically.

The RT-QA framework and data are released as an open infrastructure to accelerate the development of reliable, real-time AI assistants.