🤖 AI

Lomekwi: Resource-Bounded Tool Discovery in LLM Agents

This paper proposes a cognitive science-inspired framework that decomposes tool discovery into curiosity, recognition, and efficiency, revealing that recognition capabilities inversely scale with model size through analysis of combinatorial games and real-world task environments.

Original authors: Roshan Klein-Seetharaman, Daniel Wang, Andrew Xu

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Roshan Klein-Seetharaman, Daniel Wang, Andrew Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a robot learn to survive in a new world. For a long time, scientists have been testing these robots by handing them a toolbox and asking, "Can you use this hammer to fix the wall?" or "Can you use this wrench to tighten that bolt?" If the robot fixes the wall, we say it's smart. But this is like testing a chef by only asking them to use a knife they were just handed, never seeing if they would ever think to grab one themselves. In the real world, and in the wild, being smart isn't just about using tools you already know; it's about looking at a pile of random junk and realizing, "Hey, if I stick these two things together, I can make a tool!" This paper dives into that missing piece: the moment of discovery. It asks a simple but tricky question: As robots (specifically, AI language models) get bigger and smarter, do they actually get better at inventing their own tools, or do they just get better at using the ones they are told about?

The researchers behind this study, working with a team from Yale and Sea12 Technologies, decided to stop just measuring if the robot wins the game. Instead, they broke the game down into three distinct steps, like a recipe for invention. First, there's Curiosity: Does the robot even look for the parts it needs? Second, there's Recognition: Once it has the parts, does it realize, "Oh, I can combine these to make a machine"? And third, there's Efficiency: Once it builds the machine, can it actually use it to win? They built a special digital playground called "Lomekwi" (named after a real place where ancient humans first used stone tools) to test this. In this game, the robot has to open locked doors. It can either try to brute-force every single door until it finds the keys (a slow, boring grind), or it can find specific shards, combine them into a machine, and use that machine to open the doors instantly.

Here is where things get weird and counter-intuitive. You might expect that the bigger, more powerful AI models would be the best at everything: finding the parts, realizing how to build the machine, and using it. But the paper suggests something surprising happens. While the biggest models are great at the "grind" (solving problems by hand) and very efficient once they have a tool, they seem to lose their Recognition. It's as if the smartest models get so confident in their own ability to solve problems by brute force that they stop looking for shortcuts. They ignore the shiny new machine they could build because they think, "I can just do this myself."

In fact, the researchers found that as the models got larger, they actually became worse at spotting the opportunity to build a tool. In one experiment, a smaller model was much more likely to discover the tool recipe than a massive, super-smart model. The big models were so busy trying to solve the puzzle with their own "muscle" that they missed the "brain" solution. However, the paper also suggests this might be part of a bigger pattern. When the researchers gave the big models a little hint that a tool might exist, the big models suddenly woke up and started looking again, showing that they weren't incapable, just overconfident.

So, what does this all mean? It turns out that being a "tool user" and being a "tool inventor" are two very different skills. A model can be a genius at using a tool once it's handed to it, but a total flop at realizing it needs one in the first place. The paper suggests that as AI gets bigger, we might be seeing a dip in this creative spark of discovery, not because the AI is getting dumber, but because it's getting too sure of its own ability to grind through problems without help. It's a reminder that sometimes, the smartest thing you can do is stop trying to do it all yourself and look for a tool to help you out.

Technical Summary: LOMEKWI: RESOURCE-BOUNDED TOOL DISCOVERY IN LLM AGENTS

Problem Statement

Existing benchmarks for Large Language Model (LLM) tool use primarily evaluate a model's ability to utilize or construct tools when explicitly prompted or when the tool's existence is known. These benchmarks often conflate tool use with tool discovery, failing to measure whether an agent autonomously decides to invent a tool when faced with a novel problem. Furthermore, standard metrics typically report a single success rate, obscuring the specific cognitive processes involved in discovery. Drawing from cognitive science, the authors argue that tool invention is a distinct capability from tool use, and that current evaluations do not capture the "disposition" of an agent to build tools without instruction.

Methodology

Theoretical Framework: Decomposition of Tool Discovery

The paper proposes a formal decomposition of the tool discovery process into three separable probabilistic components, alongside a "grind" (brute-force) probability:

Curiosity ( $C$ ): The probability that the agent gathers the necessary components (ingredients) to build a tool ($Pr(hold)$).
Recognition ( $R$ ): The probability that the agent, having the components, recognizes the opportunity and commits to building the tool ($Pr(build | hold)$).
Efficiency ( $E$ ): The probability that the agent successfully solves the task using the built tool ($Pr(win | build)$).
Grind ( $G$ ): The probability of solving the task via brute force without a tool.

The total success probability is defined as:
$S = C \cdot R \cdot E + G$
This framework allows for the isolation of specific failure modes (e.g., an agent might be curious and efficient but fail to recognize the need for a tool).

Experimental Environments

To measure these components without contamination from pretraining priors, the authors introduce two environments:

Lomekwi (Combinatorial Vault Game):
- Task: An agent must open $n$ locked doors within a token/action budget.
- Mechanics: The agent can "grind" by examining doors to find keys (brute force) or "build" a machine by collecting specific shard types and combining them.
- Design: To suppress priors, all object labels (doors, keys, shards) are randomly relabeled per episode using single distinct letters.
- Measurement: Curiosity is measured by holding the correct shards; Recognition by building the machine; Efficiency by opening all doors.
Recognition MCP (Model Context Protocol):
- Task: A long-division problem where the agent can solve it manually (grind) or by calling one of $T$ candidate tools.
- Mechanics: Exactly one tool (the oracle) solves the problem instantly; the others are decoys returning "no result."
- Design: This environment isolates Recognition by providing all "ingredients" (the tools) upfront, removing Curiosity and Efficiency as variables.
- Conditions: Tested under a "baseline" prompt (tools exist) and a "tool-prior" prompt (explicitly stating one tool solves the problem).

Models Evaluated

The study evaluates a range of models, including Anthropic's Claude family (Haiku, Sonnet, Opus) and multiple Qwen families (Qwen2.5, Qwen3, Qwen3.5) with parameter counts ranging from 0.5B to 72B.

Key Contributions

Decomposition Framework: A formal separation of tool discovery into Curiosity, Recognition, and Efficiency, allowing for granular analysis of agent behavior.
Controlled Environments: The introduction of the Lomekwi and Recognition MCP environments, which use prior suppression (random relabeling) to measure genuine discovery rather than recall.
Inverse Scaling Discovery: Empirical evidence showing that Recognition scales inversely with model size in certain regimes, meaning larger models are less likely to discover or commit to using tools than smaller ones.
U-Shaped Scaling Hypothesis: Evidence suggesting that while recognition dips in intermediate models, it may recover in the largest models under specific conditions (tool-prior prompts), forming a U-shaped scaling curve.

Results

Inverse Scaling of Recognition

In the Lomekwi environment, while overall success ( $S$ ), Curiosity ( $C$ ), and Efficiency ( $E$ ) generally scale positively with model size, Recognition ( $R$ ) exhibits an inverse trend.

Observation: Larger models (e.g., Claude Opus, Qwen3.5 27B) built the required machine less frequently than their smaller counterparts (e.g., Claude Haiku, Qwen3.5 4B).
Cause: Analysis of the Recognition MCP environment revealed that larger models, being more capable at the underlying task (long division), chose to "grind" (solve manually) rather than explore tools. They abandoned the exploratory behavior necessary for recognition because they were confident in their manual capabilities.

The Role of Overconfidence

The paper attributes the inverse scaling to a calibration failure in intermediate-to-large models:

Small Models: Lack the capability to solve the task manually, so they are forced to probe tools out of necessity (high recognition).
Intermediate Models: Possess enough capability to attempt manual solutions but lack the reliability to succeed consistently. They become overconfident, abandon tool exploration, and fail both the manual attempt and the tool discovery.
Large Models: In the Recognition MCP "tool-prior" condition, the largest models recovered their recognition rates. They recognized the tool's existence as a "hedge" against failure, combining manual capability with tool exploration. This resulted in a U-shaped scaling curve for recognition across model families.

Efficiency vs. Success

The results demonstrate that aggregate success metrics are insufficient. A model can achieve high overall success by grinding (manual solving) while exhibiting near-zero tool discovery. This masks a deficit in the agent's ability to identify efficient, reusable strategies.

Significance and Claims

The paper claims that its findings necessitate a more nuanced view of LLM tool use, separating tool use from tool discovery.

Limitations of Current Benchmarks: Existing benchmarks that report single success rates or mandate tool creation fail to detect the inverse scaling of recognition. They conflate efficiency gains with discovery capabilities.
Cognitive Parallel: The work aligns with cognitive science literature (e.g., children using tools before inventing them; neurological dissociation of use and invention), suggesting that LLMs exhibit similar dissociations.
Calibration Failure: The inverse scaling of recognition highlights a calibration issue where models' decision to reach for tools is misaligned with their actual capability. As models become more capable, they may become less likely to seek efficient, generalizable solutions (tools) in favor of immediate, brute-force computation.

The authors conclude that while they have identified this phenomenon in controlled, parameterized environments, future work is needed to determine how these findings transfer to naturalistic, real-world tool discovery tasks where priors are ubiquitous.