💻 computer science

Hierarchical Compositionality for An Assistive AI Agent

This paper proposes a resource-efficient, interpretable AI agent architecture that leverages hierarchical compositionality and human-validated semantic features to effectively resolve object ambiguity through reasoning and user-specific adaptation, outperforming state-of-the-art data-driven baselines.

Original authors: Tianyi Fu, Mohan Sridharan

Published 2026-08-12

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianyi Fu, Mohan Sridharan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to give a friend a quick instruction in a busy kitchen: "Grab the fruit." If there's an apple, a banana, and an orange on the counter, your friend might pause. They have to guess which one you mean. Humans are great at this because we don't just look at the words; we remember what you usually eat, what you were doing five minutes ago, and what feels right for the moment. We use a kind of mental shortcut to figure it out without asking, "Which fruit?" every single time.

This is the world of Artificial Intelligence (AI), specifically the field of Human-Robot Interaction. Scientists here are trying to build robots and digital assistants that can help us in our homes. For a long time, the most popular way to build these helpers has been using massive "black box" computers called Large Language Models (LLMs). Think of these as super-smart parrots that have read almost everything on the internet. They are amazing at guessing what comes next in a sentence, but they can be messy. They often make random guesses when they encounter something new, they need huge amounts of energy to run, and they don't really "understand" why they chose an apple over a banana—they just picked the most likely one based on patterns.

The big question researchers are asking is: Can we build an AI that is smarter about how it thinks, rather than just having a bigger memory? Can we teach it to use simple, human-like logic and habits to solve problems quickly, without needing a supercomputer? This paper dives into that exact puzzle, trying to create an AI assistant that doesn't just guess, but reasons its way to the right answer using a clever mix of logic and memory.

The Problem: The "Which One?" Dilemma

Imagine you are the boss of a helpful robot in your house. You say, "Put the fruit on the table." Your kitchen has an apple, an orange, and a banana. To a human, this is easy. Maybe you just ate an apple, or maybe you're in a hurry and usually grab the banana. But to a standard AI, this is a nightmare. It sees three fruits. It doesn't know which one you want.

If the AI is too unsure, it might stop and ask, "Which fruit?" But if you have to ask that question every time you give a command, the robot becomes annoying and slow. On the other hand, if the AI just guesses, it might grab the wrong fruit, and you get frustrated.

The authors of this paper noticed that modern AI (like the big chatbots) is great at predicting words but terrible at this kind of personalized guessing. It's like a chef who knows every recipe in the world but doesn't know that you hate cilantro. They wanted to build a robot that actually learns your specific habits and uses them to solve the mystery.

The Solution: A Three-Layered Mental Ladder

Instead of letting the AI guess randomly or rely on a giant, opaque brain, the authors built a new kind of architecture (a blueprint for how the robot thinks) based on Hierarchical Compositionality. That's a fancy way of saying: "Break things down into small pieces, build them back up into groups, and then learn how those groups connect to you."

Think of it like building with LEGO bricks.

Level 0: The Bricks (Atomic Attributes)
First, the robot looks at every object in the house and breaks it down into tiny, simple features. Instead of just seeing "Apple," it sees: is red, is round, is sweet, has a stem. These are like the basic LEGO bricks. The robot gets these descriptions from a database of how humans actually describe things.

Level 1: The Pre-built Sets (Domain Concepts)
Next, the robot notices that certain bricks always stick together. It sees that is red, is round, and has a stem often appear together. It groups them into a "Concept" called "Apple-like." It does this for everything. A "Lamp" might be a group of emits light, has a bulb, and sits on a table. This is the robot's general knowledge of the world, shared by everyone.

Level 2: Your Personal Playbook (User-Specific Patterns)
This is the magic part. The robot starts watching you. It notices that every time you interact with a "Book-like" object (which has printed text and is flat), you almost always follow it up by turning on a "Lamp-like" object. It doesn't just remember "You like books"; it learns the pattern: Book → Lamp.

This is the Hierarchical Compositionality. The robot builds a ladder of knowledge:

Bricks: What things are made of.
Sets: What groups of things exist.
Your Playbook: How you specifically move from one group to another.

How the Robot Solves the Mystery

When you say, "Put the fruit on the table," the robot goes through a three-step process to figure out which fruit you mean.

Step 1: The Logic Filter (The "No-Go" Zone)
First, the robot uses strict logic (called Answer Set Programming) to cross out impossible options. If the table is already full, or if the fruit is in a locked box, it removes those from the list. It's like a bouncer at a club checking IDs.

Step 2: The Three Clues (Scoring the Candidates)
If there are still multiple fruits left (say, an apple and a banana), the robot scores them using three different "clues":

Semantic Clue (Context): What was happening before? If you just put a book on the table, the robot thinks, "Oh, this is a 'reading' context." It checks which fruit fits a reading context best.
Salience Clue (Recency): What did you just touch? If you just held the apple, the apple gets a high score because it's fresh in your mind.
Thematic Clue (Your Habits): This is where the Level 2 ladder comes in. The robot checks your personal playbook. "Does this user usually grab bananas after reading?" If yes, the banana gets a huge boost in points.

Step 3: The Decision
The robot adds up the scores. If one fruit is clearly the winner (by a big enough margin), it grabs that fruit. If the scores are too close, it asks, "Did you mean the apple or the banana?" This ensures it doesn't guess blindly.

What the Experiments Showed

The authors tested this robot in a simulated house with 66 different objects (like cups, books, lights, and fruits) and five different "users" with unique habits. They gave the robot 3,000 ambiguous commands to solve.

They compared their new robot against the "state-of-the-art" giants (like the latest Large Language Models) and some simpler versions of their own robot.

The Results:

The New Robot Won: In almost every test, their hierarchical robot was much more accurate than the big AI models, even though the big models had access to the exact same information.
The "Pattern" Power: The robot was especially good when the command was very vague (like saying "it" instead of "the apple"). In these hard cases, the robot's ability to use Level 2 patterns (your personal habits) made a huge difference. The big AI models kept guessing wrong because they didn't have that personal playbook.
Less Asking, More Doing: The new robot asked for help (clarification) far less often than the others, but when it did make a choice, it was right more often.
The "Transfer" Trick: The robot could figure out what you wanted even with new objects you'd never seen before, as long as they shared the same "bricks" (attributes) as old objects. For example, if you usually eat peaches, and the robot sees a new fruit that looks and feels like a peach, it guesses you want that too.

What the Paper Says It's NOT

It's important to note what this paper doesn't claim.

It doesn't say the big AI models (LLMs) are useless. They are just not the best tool for this specific job of personalized, low-data reasoning.
It doesn't claim to have solved all AI problems. The robot still needs to be tested on real physical robots in the real world, not just in a computer simulation.
It doesn't say the robot is "conscious." It's just using clever math and logic to mimic how humans use shortcuts.

The Takeaway

This paper suggests that we don't always need bigger, more expensive AI brains to make better assistants. Sometimes, we just need a smarter way to organize what we already know. By building a "mental ladder" that connects simple facts to complex groups, and then learning how you specifically move up and down that ladder, an AI can become a much better, more intuitive helper. It's like teaching a robot not just to read the dictionary, but to understand your story.

The authors found that this approach works better than the current giants in the field, especially when the robot has to guess what you want based on your unique habits. It's a step toward robots that don't just follow orders, but actually understand us.

Technical Summary: Hierarchical Compositionality for an Assistive AI Agent

Problem Statement

The paper addresses the challenge of command disambiguation for assistive AI agents operating in human environments. While modern Large Language Models (LLMs) and Vision Language Models (VLMs) demonstrate impressive capabilities, they often function as opaque, resource-intensive stochastic predictors that struggle with personalized disambiguation in novel situations. Specifically, when a human user issues an ambiguous command (e.g., "place a fruit on the table" when multiple fruits are present, or "switch it on" when multiple devices are available), data-driven models often fail to rapidly adapt to specific user preferences without extensive training data or explicit history. Existing hybrid approaches often rely on substantial human supervision or fail to transfer accumulated evidence across related entities. The core problem is how to design an architecture that leverages hierarchical compositionality and simple heuristics to resolve ambiguity based on limited interaction history, domain constraints, and user-specific habits, without requiring the massive data resources of foundation models.

Methodology

The proposed architecture resolves ambiguity through a three-stage framework that integrates symbolic reasoning, compositional representation, and heuristic scoring.

1. Knowledge Representation and Reasoning (ASP)

The system utilizes Answer Set Programming (ASP) for non-monotonic logical reasoning.

Domain Modeling: The environment is modeled using Action Language $AL_d$ , defining sorts (e.g., room, food, furniture), statics (unchanging attributes), fluents (changing states), and actions.
Feasibility Filtering: Upon receiving a command, the agent constructs an ASP program incorporating domain axioms (causal laws, state constraints, executability conditions) and the current history. It filters out candidates that violate physical constraints (e.g., an agent cannot give an object to a user in a different room) or domain logic.
Planning: Once an interpretation is selected, ASP is used to generate and execute a task plan.

2. Hierarchical Compositional Representation

To overcome data scarcity and enable knowledge transfer, the architecture represents entities through a three-layer hierarchy:

Level 0 (Atomic Attributes): Entities are represented by fine-grained, shared atomic attributes drawn from NOVA, a human-validated semantic feature norm dataset (e.g., emits_light, has_printed_text). This allows evidence to transfer between entities sharing attributes.
Level 1 (Domain-Level Concepts): Statistically significant combinations of Level 0 attributes are mined to form a concept library ( $K$ ). These concepts represent higher-level archetypes (e.g., a combination of attributes defining a "reading material"). This layer is user-independent and domain-specific.
Level 2 (User-Specific Workflow Patterns): The system mines the user's interaction history ( $H_u$ ) to identify trigger-target concept pairs. For a specific action (e.g., switch_on), the system identifies which concepts (triggers) typically precede the selection of which other concepts (targets) for that specific user. This captures personalized habits (e.g., a user who reads a book before turning on a lamp).

3. Multi-Signal Scoring and Disambiguation

When multiple feasible candidates remain after ASP filtering, they are ranked using three complementary heuristic signals:

Semantic Compatibility ( $S_{sem}$ ): Measures similarity between the candidate and the context established by the preceding action (using WordNet-based Wu–Palmer similarity).
Session Salience ( $S_{sal}$ ): Measures the recency of the candidate's mention or interaction within the current session, applying geometric decay to older references.
User-Specific Thematic Preference ( $S_{them}$ ): The core contribution, calculated by fusing:
- Concept Evidence: Frequency of the candidate's concepts in the user's history for the specific action.
- Pattern Evidence: The statistical significance (lift) and follow rate of the trigger-target concept pairs identified in Level 2.

Decision Policy: The scores are fused using weighted sums. The agent selects the top candidate only if its margin over the second-best candidate exceeds a confidence threshold; otherwise, it requests human clarification.

Key Contributions

Hierarchical Compositional Architecture: The paper introduces a novel architecture that moves beyond flat object representations. By decomposing entities into atomic attributes and recombining them into concepts and user-specific patterns, the system enables preference transfer to novel or rarely observed entities (e.g., inferring a preference for an apricot based on a history of peaches).
Integration of Heuristics and Symbolic Reasoning: The work demonstrates how ecological rationality (using simple heuristics like tallying and lexicographic search) can be effectively combined with formal logical reasoning (ASP) to achieve robust disambiguation with limited data.
User-Specific Workflow Mining: The system automatically learns user-specific behavioral patterns (Level 2) without explicit programming, allowing the agent to anticipate preferences based on the sequence of actions and concepts.
Uncertainty-Aware Clarification: The architecture employs a threshold-based policy to request clarification only when necessary, balancing the cognitive burden on the user with the need for accuracy.

Experimental Results

The system was evaluated in a physics-based simulated household environment with 66 entities, 4 rooms, and 12 actions, and 5 distinct user profiles. The evaluation involved 3,000 ambiguous commands across four levels of ambiguity (from near hypernyms to pronouns). The baselines included a data-driven model referred to in the paper as GPT-5.1 (cited as Singh et al. 2025 in the source text, which is a preprint from 2026).

Performance vs. Baselines: The proposed method significantly outperformed all baselines, including:
- LLM Baselines (GPT-5.1): Even when provided with the same full interaction history and world state, the LLM baselines (with and without Chain-of-Thought) achieved substantially lower accuracy (e.g., 30.9% average vs. 62.7% for the proposed method in noask mode) and requested clarification far more frequently.
- Ablation Studies: Removing the compositional hierarchy (using only object identities or raw attributes) or removing the workflow patterns resulted in significant performance drops, particularly under high ambiguity.
Preference Transfer: The system maintained high accuracy even for rare targets (entities with little to no direct history) by leveraging shared concepts, whereas object-level baselines failed to generalize.
User Specificity: Using a mismatched user's history degraded performance to below the level of having no history at all, confirming the system's reliance on accurate, personalized modeling.
Robustness: The method showed consistent gains across all ambiguity levels, with the largest improvements observed when ambiguity was highest (pronoun references), where workflow patterns became critical.

Significance and Claims

The paper claims that hierarchical compositionality and simple heuristics offer a viable alternative to purely data-driven approaches for assistive agents. The authors argue that:

Efficiency and Adaptability: By leveraging compositional structures, agents can rapidly adapt to specific user profiles from limited interaction history, a capability where current LLMs struggle.
Interpretability and Control: The use of ASP and explicit concept hierarchies provides a transparent reasoning process, allowing for auditing and revision of the agent's knowledge, unlike the "black box" nature of deep networks.
Ecological Rationality: The work supports the view that heuristics, when grounded in core capacities and domain structure, can outperform complex statistical models in open-world, uncertain environments by reducing variance and focusing on relevant signals.

The authors position this work as a step toward a long-term objective of developing assistive agents that use modern AI tools (like LLMs) as one component within a broader, principled architecture based on iterative refinement and interactive learning.