🤖 AI

LLM within MCP Matters: Measuring Inefficient Resource Utilization Driven by LLMs

This study reveals that Large Language Models often ignore efficiently embedded reference data in Model Context Protocol (MCP) server instructions in favor of search tools due to behavioral preferences, necessitating explicit host-level mechanisms to prioritize instruction context over tool selection.

Original authors: Minhan Cho, Soyoung Park, Kihyeon Jeong, Byeongkyu Jeon, Daejin Choi, Jinyoung Han

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Minhan Cho, Soyoung Park, Kihyeon Jeong, Byeongkyu Jeon, Daejin Choi, Jinyoung Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your smart assistant doesn't just chat with you, but also has a superpower: it can instantly call upon a vast library of tools and databases to solve your problems. This is the exciting frontier of Large Language Models (LLMs) working with tools. Think of an LLM as a brilliant, curious student who knows a lot but sometimes needs to look things up. To help this student, we have a new standard called the Model Context Protocol (MCP). You can think of MCP as a universal remote control that lets the student's "host" (the app running the student) hand over specific tools, data, and a set of instructions on how to use them.

The big question researchers are asking is: Does the student actually listen to the instructions? Sometimes, the instructions say, "Hey, I've already written the answer you need right here on this page; just read it!" But other times, the instructions also say, "Oh, and by the way, there's a search button you can press if you want." It turns out that even the smartest students might get distracted by the shiny search button, ignoring the answer sitting right in front of them. This paper dives into that exact scenario to see if the "search habit" is stronger than the ability to read the instructions, and if so, how we can fix it.

The Great "Read the Menu" vs. "Call the Waiter" Experiment

In this study, the researchers set up a digital kitchen called LexLink, a server designed to help people find Korean laws. Imagine this server is a restaurant menu. Usually, if you want a specific dish (a law), you have to ask the waiter (the search tool) to go to the kitchen, find the recipe, and bring it back. This takes time and effort.

However, the restaurant owner (the server) decided to be smart. They printed a "Daily Specials" list right on the table (the server instructions) that included the top 20 most popular dishes and their exact order numbers. The rule was simple: If you want a dish on this list, just tell the waiter the order number directly. Don't bother asking them to search the kitchen! This is supposed to be faster and cheaper.

The researchers then invited 24 different AI students (from families like Claude, Gemini, and GPT) to order from this menu. They ran a massive experiment with 54,000 trials to see what would happen. Would the AI students read the "Daily Specials" list and order directly? Or would they ignore the list and press the "Search" button anyway?

The Shocking Discovery: The Search Button is Too Tempting

The results were a bit of a wake-up call. When the researchers removed the search button entirely, forcing the students to use the table list, 23 out of 24 AI models did a fantastic job, reading the list and ordering correctly 98% to 100% of the time. This proved that the students could read the instructions; they just didn't want to when a search button was available.

But when the search button was present, the story changed dramatically. 9 out of the 24 models dropped their success rate to below 15%. They completely ignored the "Daily Specials" list and kept hitting the search button, even though the answer was right there on the table. The researchers call this a "behavioral preference." It's not that the AI is too dumb to read; it's that it has a habit of reaching for the tool it knows best, even when it's the inefficient choice. It's like a kid who knows the answer is on page 5 but keeps asking the teacher to open the book to page 5 anyway.

Interestingly, being "newer" or "smarter" didn't guarantee better behavior. Some of the very latest models actually got worse at following this specific rule than their older siblings. The study found that model recency does not predict who will be a good listener in this situation.

Can We Train Them to Listen? (The Magic Formula)

The researchers then tried to fix this bad habit by changing the instructions on the table. They tested three different tricks:

The Bossy Header (B): A big, bold warning saying, "MUST use the list! DO NOT search!"
The Example (C): Showing a picture of someone doing it right and someone doing it wrong.
The Hint (D): Changing the description of the search button to say, "Check the list first before using me."

They mixed and matched these tricks in every possible combination. The results showed that no single trick worked for everyone. In fact, for some AI models, using just the "Bossy Header" made them perform worse (dropping to 0% in one case!). It's like how a loud shout might scare one student into working, but make another student freeze up.

However, when they combined all three tricks together, the results were amazing. 20 out of the 24 models jumped back up to 86% or higher success rates. This suggests that while you can't just yell at an AI to fix its habits, a well-rounded approach that combines warnings, examples, and hints can get most of them back on track.

What This Means for the Future

The main takeaway is that simply giving an AI a tool and a set of instructions isn't enough. The AI might have a "search reflex" that overrides the instructions, leading to wasted time and resources. The paper argues that the apps running these AIs (the "hosts") need to step in. Instead of hoping the AI will read the instructions, the app should force the AI to check the instructions before it's even allowed to pick a tool.

Until that happens, developers have to be careful. They can't just write one set of instructions and expect it to work for every AI. They need to test their instructions against the "weakest" models they expect to use, and they should use a mix of examples and hints rather than just bossy commands. The study shows that while the problem is real, it's also solvable with the right recipe of instructions.

Technical Summary: LLM within MCP Matters: Measuring Inefficient Resource Utilization Driven by LLMs

Problem Statement

The Model Context Protocol (MCP) standardizes how servers expose data and tools to Large Language Models (LLMs). A prevalent design pattern involves embedding frequently used reference data (e.g., identifier lookup tables) directly into Server Instructions—the system-prompt text provided to the client LLM. Theoretically, this allows the model to bypass tool calls (such as search) for known entries, reducing token usage and latency.

However, the efficacy of this pattern remains unverified at scale. While servers provide these instructions, the client LLM must choose to follow them over available tool schemas. Prior evidence suggests low compliance with server instructions and failures in instruction hierarchies. This paper investigates whether client LLMs actually consume instruction-embedded data or if they exhibit a behavioral preference to invoke search tools even when the necessary data is explicitly provided in the context.

Methodology

The authors conducted a large-scale empirical study using LexLink, a production MCP server for Korean legal information retrieval.

Testbed: The server exposes 26 tools. For the 20 most frequently requested laws, the server instructions embed a table mapping law names to IDs. An efficient client should use eflaw_service with the cached ID directly, rather than calling eflaw_search to resolve the ID first.
Models Evaluated: 24 LLMs across three providers (9 Anthropic Claude, 6 Google Gemini, 9 OpenAI GPT), spanning multiple generations and capability tiers.
Scale: 54,000 trials (24 models × 9 conditions × 5 queries × 50 rounds).
Metric: Hit Ratio ( $\phi$ ): The fraction of trials where the model's first tool call is the correct eflaw_service with the cached ID. Any other first call (including search or wrong ID) is a miss.
Experimental Conditions:
- Baseline: Standard server instructions with the embedded table and the search tool available.
- Diagnostic (no_s): The search tool is removed entirely. This isolates capability (can the model read the table?) from preference (does it choose the table over the tool?).
- Factorial Interventions ( $2^3$ ): Three instruction-level modifications were tested individually and in combination:
  - B (Directive): A "CRITICAL" header mandating direct use of cached IDs.
  - C (Few-shot): Examples contrasting correct vs. incorrect tool patterns.
  - D (Tool Hint): Modifying the search tool's description to warn users to check the instruction table first.

Key Results

Capability vs. Preference:
- Under the no_s condition (search tool removed), 23 of 24 models achieved a hit ratio $\ge$ 98%, proving they possess the capability to read and utilize instruction-embedded data.
- Under the baseline condition (search tool present), 9 models dropped below 15%.
- Conclusion: The failure is driven by a behavioral preference to invoke the search tool rather than a lack of capability to parse instructions. The mere presence of a search tool overrides the embedded data for many models.
Non-Monotonic Performance:
- Efficiency does not correlate with model recency or generation. Notable regressions were observed:
  - GPT-4.1 (99%) $\to$ GPT-5 (1%).
  - Gemini 2.5-pro (100%) $\to$ Gemini 3-pro (2%).
  - Claude Opus 4.5 (100%) $\to$ Opus 4.6 (63%).
- This suggests that post-training on agentic benchmarks may inadvertently reinforce tool-calling policies that ignore context data.
Factorial Intervention Analysis:
- Individual interventions (B, C, or D) showed high variance and could backfire. For example, applying only the Directive (B) caused GPT-5.2's performance to drop from 5% to 0%.
- Interaction Effects: Strong interactions exist between factors. The combination of all three (BCD) restored performance to $\ge$ 86% for 20 of 24 models.
- Single-factor designs would have missed these critical interactions and the potential for interventions to degrade performance.

Significance and Claims

The paper argues that per-server prompt engineering is a workaround rather than a fix. Because individual interventions can fail or reverse depending on the model family, relying solely on instruction tuning is brittle.

The authors propose three implications:

Joint Capability Assessment: MCP server capabilities cannot be evaluated in isolation; they depend heavily on the client LLM's behavior.
Host-Side Mechanisms: MCP host applications (e.g., Claude Desktop, Cursor) should provide explicit mechanisms to prioritize server instructions in the client LLM's deliberation process (e.g., pre-injecting instructions as system-priority directives or enforcing a "consult-instructions-first" check before tool selection).
Server Developer Guidance: Developers should include few-shot examples (which showed positive or negligible effects across all models) and combine multiple intervention types (B, C, and D) for broader coverage, while validating against the weakest expected client models.

The study is limited to a single server domain (legal lookup) and single-turn interactions. While an expanded study on four additional MCP servers (geographic, academic, database, messaging) is underway, the authors explicitly state that its preliminary results are cited as an outlook rather than as evidence for the findings reported here.