🤖 AI

Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systems

This paper proposes and evaluates a novel multi-agent architecture that integrates Theory of Mind, BDI-style internal beliefs, and symbolic solvers to investigate how these cognitive mechanisms interact with varying LLM capabilities to enhance collaborative decision-making and system accuracy in resource allocation tasks.

Original authors: Adam Kostka, Jarosław A. Chudziak

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adam Kostka, Jarosław A. Chudziak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of friends trying to run a busy city together. They have to deliver food, medicine, and security to different neighborhoods before the neighborhoods get sick. This is the basic setup of the research paper you shared.

The authors are asking a big question: If we give these AI "friends" special brain upgrades—like the ability to guess what others are thinking (Theory of Mind) and a strict internal checklist to make sure they aren't crazy (Internal Beliefs)—will they work better together?

Here is the breakdown using simple analogies:

1. The Setup: The City Game

Think of the AI agents as delivery drivers in a video game.

The Goal: Keep four city districts healthy. If a district runs out of food, medicine, or security, its "health" drops.
The Problem: The drivers can't see everything. They have to talk to each other to figure out who is bringing what and where.
The Players: The researchers tested different "drivers" (AI models like ChatGPT-4o, Llama, Claude) to see which ones are the best team players.

2. The Two "Brain Upgrades"

The researchers tried giving the AI two specific tools to help them coordinate:

Tool A: Theory of Mind (ToM) – "The Mind Reader"
- What it is: This is the ability to think, "I know that Bob is a food driver, and he is probably heading to the hospital because he saw it was low on food."
- The Analogy: It's like playing poker. You aren't just looking at your own cards; you are guessing what your opponent is holding and what they plan to do next.
- The Goal: To stop two drivers from showing up at the same house with the same item while another house gets nothing.
Tool B: Internal Beliefs (IB) + Logic Check – "The Fact-Checker"
- What it is: Before an AI makes a move, it writes down its own thoughts in a strict, logical format (like a math equation) and runs it through a "logic solver" (a computer program called ASP).
- The Analogy: Imagine you are about to drive, and you say, "I am at the store, I have 5 apples, and I need to go to the park." The Logic Check is like a strict teacher who instantly yells, "Wait! You said you have 5 apples, but the math says you only have 3! You can't go to the park yet!"
- The Goal: To stop the AI from hallucinating (making things up) or forgetting where it is.

3. The Experiment: Mixing and Matching

The researchers ran the city game with different combinations:

No Upgrades: Just the raw AI driver.
Mind Reader Only: The AI guesses what others are doing but doesn't check its own math.
Fact-Checker Only: The AI checks its own math but doesn't guess what others are doing.
Both Upgrades: The AI guesses what others are doing AND checks its own math.

4. The Results: It's Complicated!

Here is the surprising part: Giving the AI more "brain power" didn't always make it smarter.

The Superstars (ChatGPT-4o & Claude): These models were already so smart that adding the extra tools helped them stay consistent, but they were already doing a great job without them. They handled the "Mind Reading" and "Fact Checking" perfectly.
The Strugglers (Smaller Models like Llama or ChatGPT-3.5): For these models, the extra tools were a mixed bag.
- Sometimes, the "Mind Reader" tool helped them guess better.
- Other times, the "Fact Checker" confused them. It was like giving a student a calculator that was too complicated; they spent so much time trying to fix their math that they forgot to actually deliver the food.
- The "Cognitive Overload" Metaphor: Imagine a small car trying to carry a heavy engine. Sometimes the engine helps the car go faster. Other times, the car is so heavy it can't move at all. The smaller AI models got "overloaded" by trying to do too much thinking at once.

5. The Big Takeaway

The main lesson from this paper is that more features don't automatically mean a better team.

One size does not fit all: A "Mind Reader" tool works great for a super-smart AI, but it might confuse a simpler one.
The "Checklist" helps, but costs time: Checking your own logic prevents mistakes, but it takes extra computing power. If the AI is already slow, this might make it slower.
Teamwork is hard: Even with these fancy tools, getting AI agents to coordinate perfectly in a changing world is still very difficult.

In summary: The researchers built a cool new system where AI agents can "read minds" and "check their own math." They found that while this is a great idea, it only works well if the AI is smart enough to handle the extra thinking. If the AI is too simple, the extra tools might just make it trip over its own feet.

1. Problem Statement

While Large Language Models (LLMs) have revolutionized Multi-Agent Systems (MAS) by enabling natural language interaction and planning, they face significant challenges in dynamic, collaborative environments:

Inconsistency: LLMs often produce inconsistent reasoning or hallucinations, leading to suboptimal coordination.
Lack of Cognitive Structure: Simply adding LLMs does not guarantee effective collaboration. Existing MAS frameworks often rely on rigid rules or lack mechanisms for agents to model each other's intentions (Theory of Mind) or maintain consistent internal state representations (Belief-Desire-Intention).
The Gap: The synergistic integration of Theory of Mind (ToM), Internal Belief (IB) mechanisms (based on the BDI model), and Symbolic Logic Verification remains underexplored. It is unclear how these cognitive modules interact with different LLM architectures to influence collective decision-making and system accuracy.

2. Methodology

A. Task Environment

The authors designed a dynamic resource allocation simulation set in a virtual city with four districts ( $d_1$ to $d_4$ ).

Objective: Three agents ( $FOOD\_AGENT$ , $MEDICAL\_AGENT$ , $SECURITY\_AGENT$ ) must coordinate to supply resources to districts to prevent health decline.
Mechanics: Districts consume resources over time. If resources drop below thresholds, district health decreases. Agents can MOVE to adjacent districts or SUPPLY_RESOURCE.
Metrics: System performance is measured by the average final health of the non-supply districts.

B. Proposed Architecture

The system introduces a novel agent architecture integrating three core cognitive components:

Theory of Mind (ToM): Agents maintain a "Beliefs on Others" module. They infer the roles, intentions, and likely actions of peer agents based on observable behavior and shared memory.
Internal Beliefs (IB): Agents maintain a private "Internal Beliefs" module representing their understanding of the game state (observations, plans, needs) in a logic-friendly format.
Logical Verification (Symbolic Solver): The agent's natural language internal beliefs are translated into Answer Set Programming (ASP) using the Clingo solver.
- Process: The solver checks for logical consistency (e.g., contradictions between location and inventory).
- Feedback Loop: If inconsistencies are detected, the system generates a natural language explanation of the error, allowing the LLM to reformulate its beliefs (up to 3 attempts) before executing an action.

C. Experimental Setup

Models Tested: ChatGPT-3.5-Turbo, ChatGPT-4o, ChatGPT-4o-mini, Meta Llama 3.1 8B, and Claude 3.5 Sonnet.
Configurations: Four distinct setups were tested for each model:
1. Base Case: No ToM, No IB.
2. ToM Only: Agents reason about others but lack internal belief verification.
3. IB Only: Agents use internal beliefs with ASP verification but do not model others.
4. ToM + IB: Full integration of both mechanisms.
Protocol: Agents communicate via a shared memory (for coordination) and private memory (for internal reasoning). Simulations ran for 7 turns per agent.

3. Key Contributions

Novel Architecture: A multi-agent framework that seamlessly integrates ToM, BDI-style internal beliefs, and ASP-based logical verification to augment collaborative intelligence.
Empirical Evaluation: A comprehensive analysis of how these cognitive mechanisms interact with varying LLM capabilities (from 8B parameter models to advanced commercial models).
Insight on Cognitive Load: The study demonstrates that adding cognitive mechanisms is not universally beneficial; the efficacy depends heavily on the underlying LLM's reasoning capacity and the complexity of the task.

4. Results and Analysis

Performance Variability

High-Performance Models (ChatGPT-4o, Claude 3.5 Sonnet): These models consistently achieved high district health scores across all configurations. They successfully integrated ToM and IB without performance degradation, indicating robust reasoning capabilities that can handle the added cognitive load.
Intermediate Models (ChatGPT-4o-mini): Showed mixed results. While capable, the addition of ToM/IB sometimes increased variability (wider confidence intervals), suggesting the cognitive overhead occasionally confused the model.
Lower-Capacity Models (ChatGPT-3.5-Turbo, Llama 3.1 8B):
- ChatGPT-3.5-Turbo: Showed slight improvements with ToM/IB, suggesting these mechanisms can scaffold reasoning for weaker models.
- Meta Llama 3.1 8B: Exhibited significant instability. While ToM alone helped, the combination of ToM + IB often led to performance drops, indicating the model struggled to process multiple simultaneous reasoning streams (modeling others + self-verification).

Key Findings

The "Two-Edged Sword" Effect: Integrating ToM and IB can enhance coordination by enabling better prediction of peer actions and structured decision-making. However, for models with limited reasoning capacity, these mechanisms introduce cognitive load, leading to false assumptions, logical inconsistencies, and miscoordination.
Role of Logical Verification: The ASP solver successfully prevented some logical errors (e.g., moving to disconnected districts), but its effectiveness was limited by the quality of the natural language translation from the LLM.
Communication Bottleneck: The sequential communication protocol (shared memory) was effective but introduced latency and potential information flow issues in complex scenarios.

5. Significance and Future Directions

Implications for AI Design: The paper establishes that "more cognitive features" does not equal "better performance." System architects must carefully match cognitive mechanisms (ToM, BDI, Logic) to the specific reasoning capabilities of the chosen LLM.
Hybrid Intelligence: The work validates the potential of Neural-Symbolic AI, where LLMs handle natural language and high-level planning, while symbolic solvers (ASP) ensure logical consistency and safety.
Future Work:
- Developing learned models of agent behaviors rather than static ToM prompts.
- Exploring hybrid reasoning approaches to reduce the cognitive load on LLMs.
- Optimizing communication protocols to move beyond sequential sharing.
- Scaling the system to larger numbers of agents and more complex task environments (e.g., disaster management, healthcare).

In conclusion, this research provides a critical empirical foundation for building robust LLM-based Multi-Agent Systems, highlighting that successful collaboration requires a tailored balance between model capacity, cognitive architecture, and logical verification.