🤖 AI

How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?

This paper presents an empirical study evaluating the performance of tool-augmented LLM agents on 243 expert-curated, real-world energy market analytics tasks—ranging from live data retrieval to quantitative modeling—using a multi-dimensional scoring protocol to assess how model capabilities and domain-specific tools interact in high-stakes professional settings.

Original authors: David Akinpelu, Akintonde Abbas, Rereloluwa Alimi, Ayodeji Lana

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David Akinpelu, Akintonde Abbas, Rereloluwa Alimi, Ayodeji Lana

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the energy sector (electricity markets, regulations, and power grids) as a massive, chaotic library where the books are constantly changing, the prices are written in invisible ink, and the rules are written in a language only a few experts understand.

For a long time, Artificial Intelligence (AI) has been like a very smart student who can memorize the library's catalog perfectly but can't actually go find the books, read the fine print, or do the math to figure out if a power plant is making money.

This paper, titled "How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?", is a report card on a new kind of AI student. These students aren't just memorizing facts; they are equipped with a toolbox (like a set of keys, calculators, and search engines) that lets them go out, fetch live data, read real regulations, and run complex financial models.

Here is the breakdown of what the researchers did and what they found, using simple analogies:

1. The Test: A "Real-World" Obstacle Course

The researchers didn't just ask the AI simple trivia questions like "What is the capital of Texas?" (which is easy to memorize). Instead, they built a 243-question obstacle course designed by real energy experts with decades of experience.

The course had three types of challenges:

The Detective (Data Retrieval): "Go find the electricity price in Houston for last Tuesday at 2 PM."
The Lawyer (Knowledge Interpretation): "Read this 50-page government rulebook and tell me exactly how much a battery company has to pay to connect to the grid."
The Accountant (Quantitative Modeling): "If I build a battery here, and electricity prices swing like this for 15 years, will I make a profit? Calculate the exact number."

The tasks ranged from Easy (finding a number) to Hard (solving a multi-step financial puzzle with strict rules).

2. The Contestants: Seven AI "Brains"

They tested seven different AI models (some made by big tech companies like OpenAI and Google, and some open-source ones).

The Setup: They gave these AIs a "backpack" of nine specific tools. These tools included live connections to electricity markets, databases of utility rates, and code runners to do the math.
The Rule: The AIs had to use these tools to solve the problems, not just guess based on what they remembered.

3. The Scoring System: Not Just "Right or Wrong"

The researchers didn't just check if the final number was right. They graded the AIs on three things, like a strict teacher:

Approach (Did they think like a pro?): Did they pick the right tools and use them in the right order? If an AI got the right answer but guessed the data instead of looking it up, they lost points.
Accuracy (Did they get the facts right?): Was the number correct? Did they use the right time zone and the right state?
Source Validity (Did they cheat?): Did they cite the real document they read, or did they make up a fake rulebook?

4. The Results: The Good, The Bad, and The "Almost"

Here is what happened when the AIs took the test:

The "Smartest" Students: The closed-source models (like Gemini-3.1-Pro and GPT-5.2) performed the best, getting about 60% of the answers correct. The best open-source model (Kimi-K2.5) got about 49%.
- The Catch: Even the best student missed nearly half the questions. This means AI is helpful, but it's not ready to replace human experts yet.
The "Planning vs. Doing" Gap: Interestingly, the AIs were actually quite good at planning (scoring high on "Approach"). They knew which tool to pick. But they often failed at executing the plan perfectly (getting the wrong number or the wrong date). It's like a chef who knows exactly which ingredients to buy but burns the steak while cooking it.
The "Hallucination" Problem: The AIs were terrible at citing their sources. They often forgot to say, "I found this in the 2024 rulebook," or they made up fake document names. In the real world, if you can't prove where your data came from, you can't trust the answer.
The "Tool" Effect: When the researchers took away the tools and asked the AIs to solve the problems from memory, the scores dropped by half. This proves that for energy tasks, having the tools is more important than just having a smart brain.
The "Memory" Limit: Some models failed because the questions were too long and complex, filling up their "short-term memory" (context window). They forgot the first step of the problem by the time they got to the last step.

5. The Big Takeaway

The paper concludes that AI agents are powerful assistants, but they are not yet independent experts.

Think of it like a junior analyst who has a super-fast computer and access to every library in the world. They can do the heavy lifting and find the data quickly. However, they still need a senior expert (a human) to double-check their work, make sure they didn't make up a rule, and verify the final math.

The researchers released all their test questions, the tools, and the results to the public so other scientists can try to build better "students" for the future. They plan to expand this test to other countries and other types of energy (like oil or gas) in the next round.

Technical Summary: How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?

Problem Statement

While agentic benchmarks have matured in domains such as finance, coding, law, and drug discovery, the energy sector remains critically underserved. Existing evaluations in energy are largely limited to static knowledge recall (e.g., multiple-choice questions) or predictive tasks like load forecasting using historical data. There is a significant gap in evaluating whether Large Language Model (LLM) agents can execute end-to-end, multi-step analytical workflows under real-world constraints. These workflows require live data retrieval from ISO/RTO markets, interpretation of complex regulatory documents, and quantitative modeling for asset optimization—tasks where errors carry material financial and operational consequences. Current benchmarks fail to assess the interaction between model capability and domain-specific tooling in high-stakes professional settings.

Methodology

The authors introduce EnergyEvals, an empirical study and evaluation framework designed to assess tool-augmented LLM agents on real-world energy analytics tasks.

1. Benchmark Dataset

The study utilizes a corpus of 243 expert-curated tasks, developed by practitioners with doctoral-level training and over 25 years of combined industry experience (e.g., from McKinsey, GE, ICF). The tasks are stratified into three capability areas and three difficulty levels (Easy, Medium, Hard):

Market Data Retrieval and Analysis (Data): Extraction and aggregation of live pricing, ancillary services, and dispatch data from ISO/RTO APIs.
Knowledge Retrieval and Interpretation (Knowledge): Navigation of regulatory dockets, utility tariffs, and interconnection manuals to answer procedural questions.
Advanced Quantitative Modeling (Quant): Multi-step financial modeling, including battery revenue estimation, IRR computation, and optimization under operational constraints.

A unique design feature is paired prompt construction, where tasks are available with and without explicit source specifications to evaluate the impact of source scaffolding.

2. Agent Architecture and Tool Suite

Agents are implemented using a ReAct framework (Thought $\to$ Action $\to$ Observation), allowing for iterative reasoning and tool invocation. The environment provides access to a configurable suite of nine domain-specific tools, including:

Live electricity market APIs for major U.S. ISOs.
Regulatory docket search and utility tariff databases.
Asset optimization models (e.g., battery revenue, renewable generation simulation).
Retrieval-augmented generation (RAG) over market documents.

3. Evaluation Protocol

Performance is assessed via a multi-dimensional protocol using an LLM-as-a-judge framework (calibrated with human expert review) across three dimensions:

Approach Correctness (1–5): Evaluates the logical sequencing of tool calls and alignment with professional analyst strategies.
Answer Accuracy / Attribute Alignment (0–1): Measures factual correctness and adherence to constraints (temporal, geographic, units). For quantitative tasks, this involves tolerance-based comparison; for qualitative tasks, it involves matching specific attributes.
Source Validity (1–5): Penalizes hallucinated citations, fabricated document versions, or reliance on inappropriate sources.

The study evaluates seven frontier LLMs (four closed-source: GPT-5.2, GPT-5-mini, Gemini-3.1-Pro, Claude Sonnet 4.6; three open-source: Kimi-K2.5, Qwen3-Max-Thinking, DeepSeek-V3.2) under low-reasoning-effort configurations to simulate cost-constrained professional environments.

Key Results

The evaluation of the seven models reveals several critical insights regarding the state of agentic systems in the energy domain:

Performance Gap: Closed-source models generally outperform open-source models. Gemini-3.1-Pro achieved the highest accuracy (62%), followed by GPT-5.2 (57%) and Claude Sonnet 4.6 (56%). The best open-source model, Kimi-K2.5, scored 49%, though it did so at a significantly lower cost (58% of GPT-5.2's cost).
Planning vs. Execution: While models generally demonstrate reasonable Approach Correctness (scoring ~3.4–3.9 out of 5), there is a distinct gap between proposing a valid analytical strategy and executing it with high accuracy. This suggests that while models can reason about how to solve a problem, reliable execution requires deeper domain-specific knowledge to structure workflows and validate data.
Source Validity Deficit: Source attribution scores were generally low across all models. Models frequently failed to cite sources or hallucinated document versions, indicating that reliable source referencing is not yet an emergent behavior and requires explicit prompting or expert guidance.
Impact of Context Windows: Models with shorter context windows (e.g., DeepSeek-V3.2, Kimi-K2.5) exhibited significantly higher failure rates (up to 20.2%) due to context limitations or clarification loops, whereas models with larger windows (400k+ tokens) maintained failure rates below 3.2%.
Tool Necessity: Removing domain-specific tools resulted in a drastic performance drop. In some cases, accuracy scores were halved when agents were forced to rely on internal knowledge rather than live tools, underscoring the critical dependency of energy analytics on external data retrieval.
Token Efficiency: Higher token usage did not correlate with better performance; some lower-performing models consumed significantly more tokens, while top performers varied in efficiency.

Significance and Claims

The paper claims that EnergyEvals is the first benchmark to rigorously evaluate tool-augmented agents on real-world energy analytics workflows, moving beyond static recall to dynamic, multi-step problem solving.

The authors assert that while tool-augmented agents can tackle specific energy tasks, expert guidance remains essential to bridge the gap between "generally good" performance and "exceptionally insightful" outcomes. The study highlights that current frontier models, even when equipped with tools, struggle with source validity and complex workflow orchestration without human-in-the-loop oversight.

The work contributes a publicly released dataset, evaluation framework, scoring code, and execution traces to support reproducibility. The authors position this as a foundational step, with future iterations planned to expand geographically beyond the U.S., cover additional energy sub-domains, and explore higher reasoning configurations.