🤖 AI

Self-Evolving Neuro-Symbolic Skills for Tool-Augmented Spatial Reasoning

The paper introduces NeSy-Spatial, a neuro-symbolic framework that enhances spatial reasoning by abstracting tool interactions into reusable, self-evolving skills that are adaptively composed and refined through a closed-loop process of execution and trajectory analysis.

Original authors: Shi-Yu Tian, Zhuo-Xia Wang, Xuan-Yi Zhu, Zhi Zhou, Xinwei Yang, Kun-Yang Yu, Ming Yang, Yang Chen, Yu-Feng Li

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shi-Yu Tian, Zhuo-Xia Wang, Xuan-Yi Zhu, Zhi Zhou, Xinwei Yang, Kun-Yang Yu, Ming Yang, Yang Chen, Yu-Feng Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a tricky puzzle, but instead of just using your brain, you have a toolbox full of magical gadgets. Some gadgets can see things you can't, others can measure distances instantly, and some can do complex math in a flash. This is the world of Large Vision-Language Models (LVLMs): super-smart computer programs that can look at pictures and read text, trying to understand the world around them. They are great at general things, like saying "that's a cat" or "the sky is blue." But when it comes to spatial reasoning—figuring out exactly how far away something is, which way it's facing, or how objects move in 3D space—they often stumble. They might guess the right answer by luck, but they struggle to do the precise math and step-by-step logic required to be truly reliable.

To fix this, scientists have started giving these AI models "tools" to use, like a calculator or a map. But here's the catch: most current methods are either too chaotic or too rigid. Some let the AI pick tools on the fly, like a kid grabbing random toys from a bin, which often leads to mistakes (like trying to measure a distance before you've even found the object!). Others use a fixed, pre-written script, like a robot that follows the exact same dance steps no matter the song, wasting time on simple tasks and failing on complex ones. The big question is: how can we teach an AI to learn from its own mistakes, build a library of smart strategies, and adapt those strategies to new problems, just like a human does?

This paper introduces NeSy-Spatial, a clever new framework that acts like a self-improving apprentice for these AI models. Think of it as a "skill gym" where the AI doesn't just memorize answers but learns how to do things. The system is built on two types of "skills": Tool-Use Skills and Geometry Skills.

Tool-Use Skills are like a recipe book for using gadgets. Instead of guessing which tool to pick next, the AI learns a structured workflow: "First, use the camera gadget to see the scene. Next, use the detector gadget to find the red ball. Finally, use the math gadget to measure the distance." These skills ensure the AI doesn't skip steps or use tools in the wrong order.
Geometry Skills are like specialized math formulas. Once the AI has the data (like the location of the red ball), it doesn't just guess the distance; it pulls out a pre-written, verified code block that knows exactly how to calculate the distance between two points in 3D space.

The magic of NeSy-Spatial is that it evolves. It doesn't just use a static list of skills; it learns and grows. When the AI solves a problem, it saves the journey. If it succeeds, it analyzes the path to see what worked well and saves that as a new "skill." If it fails, it doesn't just throw the attempt away; it looks at why it failed. Did it forget to find the object first? Did the math go wrong? It then updates its library, pruning (cutting out) the bad or useless skills and refining the good ones. It's like a video game character that levels up: every time they beat a boss or fail a trap, they learn a new move or improve an old one, making them better for the next level.

The researchers tested this system on three different challenging spatial reasoning benchmarks (MMSI, MindCube, and OmniSpatial). They found that NeSy-Spatial consistently outperformed other methods. It didn't just get the right answers more often; it also used its tools more efficiently, avoiding unnecessary steps and reducing errors. For example, on one dataset, it improved the overall accuracy significantly compared to the best existing methods. The system showed that by organizing its knowledge into reusable, self-correcting skills, it could handle complex spatial puzzles much better than models that just try to guess or follow rigid scripts.

In short, NeSy-Spatial teaches AI models to be less like a chaotic child grabbing random tools and more like a seasoned craftsman with a well-organized, ever-improving toolkit. It proves that when AI agents can learn from their own experiences, refine their strategies, and adapt to new situations, they become much more reliable at understanding the physical world.

Technical Summary: NeSy-Spatial

Problem Statement

Large Vision-Language Models (LVLMs) have demonstrated strong capabilities in general multimodal reasoning but remain unreliable for fine-grained spatial tasks. These tasks often require precise visual perception and structured geometric computation that exceed the capabilities of end-to-end generation. While tool augmentation (leveraging external vision models and executors) offers a solution, existing approaches suffer from two primary limitations:

Dynamic Planning: Methods that plan tool calls from scratch often lack explicit constraints on tool dependencies and execution order, leading to invalid sequences (e.g., invoking a Python interpreter before acquiring necessary object coordinates).
Fixed Pipelines: Methods relying on manually designed pipelines are often redundant, invoking unnecessary tools regardless of task complexity, and fail to generalize across diverse spatial reasoning scenarios.

Current agents lack the ability to accumulate reusable experience and adaptively compose it for new problems, necessitating a framework that can abstract, refine, and evolve spatial skills over time.

Methodology: NeSy-Spatial

The authors propose NeSy-Spatial, a neuro-symbolic framework designed for self-evolving spatial skills. The framework abstracts tool interactions and geometric operations into typed, executable atomic instructions and composes them into two complementary skill types: Tool-Use Skills and Geometry Skills.

1. Neuro-Symbolic Skill Formulation

The skill library ($Lib$) consists of a set of atomic instructions ( $A$ ) and high-level skills ( $S$ ), where $S = S_{tool} \cup S_{geo}$ .

Atomic Instructions: These are typed executable operations with local verifiers. They are divided into:
- Tool Atoms ( $A_{tool}$ ): Wrappers for visual tools and the Python executor.
- Geometry Atoms ( $A_{geo}$ ): Reusable code blocks for geometric transformations, numerical comparisons, and relation verification, executed internally within the Python tool.
High-Level Skills: Represented as node-edge-context structures ( $s = \{V, E, \kappa\}$ $s = {V, E, κ}$ ).
- Tool-Use Skills ( $S_{tool}$ ): Organize controller-visible tool atoms into reusable workflows, defining execution dependencies and context (e.g., object bindings, tool arguments).
- Geometry Skills ( $S_{geo}$ ): Provide reusable, verifiable implementations for Python tool calls, abstracting operations like coordinate transformation and consistency checking.

2. Inference Process

During inference, NeSy-Spatial operates in a closed-loop process consisting of four stages:

Skill Retrieval: The agent queries the skill library using the current question and state to retrieve a candidate set of relevant skills.
Action Selection: The controller selects the next action: terminating with an answer, activating a retrieved Tool-Use Skill, or calling a specific tool atom. Geometry Skills are instantiated within the Python atom rather than acting as independent controller actions.
Tool/Code Execution: Selected actions execute tool atoms or dependency-ready nodes. When a Python atom is reached, the agent instantiates a retrieved Geometry Skill to guide code construction, ensuring structured and verifiable geometric computation.
State Update: Execution outputs and verifier outcomes update the working memory, initiating the next retrieval round until a final answer is produced.

3. Self-Evolution Mechanism

The framework evolves the skill library between episodes based on a buffer of completed trajectories ( $B_t$ ). Evolution is triggered when the buffer reaches a threshold ( $N_{buf}$ ).

Trajectory Analysis: The system decomposes trajectories into operations, arguments, and outcomes to identify reusable structures. It aggregates dependency patterns to update atomic instructions and identify supported or invalid skill fragments.
Skill Fusion: Extracted atoms and fragments are fused with the existing library. This involves adding novel instructions, replacing existing implementations if they better satisfy verifiers, and editing high-level skills by adding supported fragments or removing redundant dependencies.
Skill Pruning: To prevent unbounded accumulation, the system prunes entries based on invocation history. Skills or atoms are removed if they exceed an error-rate threshold, remain inactive for a specific duration, or lack sufficient invocation support. A tenure-dependent retention factor protects long-lived entries from premature removal.

Key Contributions

Neuro-Symbolic Definition of Spatial Skills: The paper introduces a unified definition that couples tool invocation and geometric computation through typed, executable atomic instructions.
NeSy-Spatial Framework: A skill self-evolution framework that updates atomic instructions and high-level skills (Tool-Use and Geometry) through analysis, fusion, and pruning of buffered trajectories.
Empirical Validation: Extensive evaluation on three spatial reasoning benchmarks (MMSI, MindCube, OmniSpatial) demonstrating consistent improvements over base static tool-use agents and fixed-pipeline methods.

Experimental Results

The authors evaluated NeSy-Spatial using GPT-5.4 and Gemini 2.5 Pro as backbones across three datasets.

Reasoning Accuracy: NeSy-Spatial achieved the best overall accuracy in five of six backbone-dataset settings. Notably, with Gemini 2.5 Pro, it outperformed the strongest baselines by significant margins (e.g., +10.12% on MindCube).
Tool Utilization: The framework significantly improved the First-Pass Compositional Tool-Chain Success Rate (FTCS) and Python execution success rates, indicating more reliable and precise tool usage.
Evolution Dynamics: Analysis showed that the skill library expanded from a small set of seed skills (3-4) to a larger, refined set (up to 20) through online interaction. Crucially, the library size was not monotonic; pruning actively removed unreliable or inactive skills, maintaining a dynamic balance between accumulation and refinement.
Ablation Studies: Disabling skill evolution or removing either Tool-Use or Geometry skills resulted in substantial performance drops, confirming that online evolution and the complementary nature of the two skill types are essential for the framework's success.

Significance and Claims

The paper claims that NeSy-Spatial addresses the critical gap in current spatial reasoning agents: the inability to learn from experience and adaptively compose reusable procedures. By moving away from planning from scratch or relying on rigid templates, NeSy-Spatial enables agents to accumulate reusable experience, refine skill structures through failure diagnosis, and prune ineffective entries. The results suggest that self-evolving neuro-symbolic skills provide a robust pathway for improving the reliability and efficiency of tool-augmented spatial reasoning, particularly in scenarios requiring precise geometric computation and multi-step inference. The authors note that current evaluation is limited to selected benchmarks and tool environments, with future work intended to examine broader tasks and transfer to unseen settings.