🤖 AI

Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints

This paper presents a neurosymbolic architecture that enables an agent to efficiently learn and assemble novel structures under previously unseen semantic constraints by integrating natural language communication of those constraints with visual observations and task demonstrations.

Original authors: Jonghyuk Park, Alex Lascarides, Subramanian Ramamoorthy

Published 2026-08-17

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonghyuk Park, Alex Lascarides, Subramanian Ramamoorthy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

=== SUMMARY ===
Imagine you are building a massive, intricate castle out of LEGO bricks. You have the bricks, you have the instructions for how they snap together physically, and you have a robot arm to do the stacking. But here's the twist: the robot doesn't know the rules of the castle. It knows that a red brick fits on a blue one, but it doesn't know that in this specific kingdom, "Red bricks are only for the King's tower, never for the dungeon." If the robot tries to build a dungeon with a red brick, it might be physically possible, but it's a "semantic" disaster—it breaks the story of the castle. This is the challenge of neurosymbolic AI, a field trying to teach robots to combine the "gut feeling" of seeing the world (neural networks) with the "logical rules" of how things should be (symbolic reasoning). Usually, robots learn by watching you do a task once or twice. But what if you introduce a brand new type of brick or a new rule halfway through the game? Can the robot learn on the fly, not just by watching your hands, but by listening to your words?

This paper, titled "Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints," explores exactly that scenario. The researchers built a simulated world where a robot tries to assemble toy trucks. The catch? The robot starts with zero knowledge of what a "dump truck" is, what a "dumper" looks like, or the rule that "dump trucks can't have yellow parts." As the robot tries to build, it makes mistakes. A human teacher then steps in, not just to show the robot how to move its arm, but to explain the logic of the mistake. The team tested different ways of teaching: showing the robot a picture, giving it a label like "this is a cabin," or giving it a full sentence rule like "All dump trucks must have a dumper."

The results, found in their computer simulations, suggest that the most effective way to teach the robot is through those full-sentence rules. When the teacher simply said, "Use this instead of that," the robot learned slowly and made many mistakes. But when the teacher explained the general rule ("Coloured parts of a dump truck must not be yellow"), the robot learned much faster and assembled the trucks with far fewer errors. It turns out that giving the robot a reusable "rule of thumb" is like handing it a reference guide for the future, rather than just correcting a single move. The study shows that while robots are getting better at seeing and grabbing, they really need our words to understand the meaning behind the assembly, especially when they encounter parts and rules they've never seen before.

The Story of the Talking Robot Builder

Picture a robot named "Builder-Bot" sitting at a table covered in colorful, blocky toy parts. Its mission? To build specific types of toy trucks, like fire trucks, missile trucks, or dump trucks. But Builder-Bot is a bit of a blank slate. It hasn't been trained on these specific trucks yet. It doesn't know what a "dumper" is, it doesn't know that a "fire truck" needs a ladder, and it certainly doesn't know the secret rule that "fire trucks must have red fenders."

In the real world, if you taught a robot to build a truck, you might just show it a video of you doing it once or twice. The robot would watch your hands, memorize the movements, and try to copy them. This is called learning from demonstrations. It works great for the physical stuff: "Pick up the red block, move it here, snap it there." But it's terrible for the rules. If you show the robot building a fire truck with a red fender, and then later ask it to build a dump truck, the robot might accidentally put a yellow fender on the dump truck. Physically, the yellow fender fits just fine. But semantically, it's wrong because dump trucks have different rules. The robot can't guess the rule just by watching you; it needs to be told.

This is where the paper's "neurosymbolic" approach comes in. Think of the robot's brain as having two distinct rooms.

The Vision Room (Neural): This is where the robot looks at the table. It sees a blob of pixels and says, "That looks like a cabin, I'm 80% sure." It's good at guessing what things are based on what it has seen before.
The Logic Room (Symbolic): This is where the robot keeps its rulebook. It holds statements like "If it's a fire truck, it must have a ladder." This room is perfect for checking if a plan makes sense, but it's blind to the world unless the Vision Room tells it what it's looking at.

The researchers wanted to see how these two rooms could talk to each other when the robot encounters something totally new. They set up a simulation where the robot tries to build trucks while chatting with a human teacher. The teacher can help in a few different ways, and the researchers tested four different "teaching styles" to see which one helped the robot learn the fastest.

The Four Teaching Styles

The researchers compared four ways the teacher could interact with Builder-Bot:

The Silent Observer (NoLabels+CaseMemory): The teacher doesn't use any specific words like "cabin" or "dumper." They just point and say, "Use this one, not that one." The robot has to guess what the parts are called and remember the specific correction for that exact moment. It's like trying to learn a new language where the teacher only points at things without naming them.
The Labeler (LabelsOnly): The teacher gives the robot the names. "This is a cabin. This is a dumper." The robot learns the vocabulary and can recognize the parts, but it doesn't get any rules about how they should be used.
The Corrector (Labels+CaseMemory): The teacher gives the names and remembers specific mistakes. "Remember, for a dump truck, use the red cabin, not the blue one." The robot learns the names and stores a list of "don'ts" for specific situations, but it doesn't learn the general rule behind them.
The Rule-Giver (Labels+TeacherRules): This is the full package. The teacher gives the names and explains the general rules. "A dump truck must have a dumper," or "Coloured parts of a dump truck must not be yellow." The robot takes these sentences and turns them into strict logical rules in its Logic Room.

The Big Discovery: Rules Beat Examples

When the researchers ran the simulation 30 times with 40 different assembly challenges each, a clear pattern emerged.

The robot that only got labels and specific corrections (Style 3) learned about as well as the one that only got labels (Style 2). They both did better than the silent observer, but they still made a lot of mistakes. Why? Because when the robot saw a new part or a new truck type, it had to guess based on its memory of specific past mistakes. It was like trying to solve a puzzle by remembering every single time you put a piece in the wrong spot, rather than understanding the picture on the box.

However, the robot that got the general rules (Style 4) was a standout. It made significantly fewer mistakes and learned much faster. The key finding here is that the robot didn't just get better at seeing the parts; it got better at planning. When the robot heard, "Dump trucks have a dumper," it didn't just memorize that one truck. It updated its internal rulebook. The next time it had to build a dump truck, its Logic Room immediately checked: "Do I have a dumper? No? Then I can't build this truck yet."

The paper suggests that this "semantic constraint" learning is the key factor. By communicating the rules of the world in natural language, the teacher gave the robot a tool it could use over and over again. It didn't have to relearn the lesson every time a new truck appeared. The robot could take the rule "Fire trucks need red fenders" and apply it to any fire truck it encountered, even if it had never seen that specific red fender before.

Why This Matters (and What It Isn't)

It's important to note that this isn't a robot that can go out and build a real car in a garage tomorrow. The experiments happened entirely in a computer simulation with toy trucks. The robot's "eyes" were perfect (it knew exactly where every piece was), and the teacher followed a strict script. The researchers are careful to say this is a step toward making robots more adaptable, not a solved problem.

But the implication is exciting. In the future, if you want a robot to help you build furniture or assemble a complex kit, you won't need to spend hours filming yourself doing it. You might just be able to say, "Hey, the legs of this table need to be the heavy ones, not the light ones," and the robot will instantly understand the rule and apply it to the next table you ask it to build.

The paper shows that while robots are getting great at seeing and moving, they are still learning how to listen to the logic of our language. By combining the robot's ability to see with its ability to reason about rules, we can teach them to handle the unexpected, turning them from rigid copycats into flexible, learning partners. The robot didn't just learn to build a truck; it learned how to learn the rules of the game.

Technical Summary: Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints

Problem Statement
The paper addresses the challenge of robotic assembly in open-ended environments where agents encounter semantic constraints post-deployment. While prior work effectively handles geometric constraints (part shapes, join poses) and assembly orders via task demonstrations, it often fails to account for semantic constraints: domain conventions that render a physically feasible join invalid (e.g., "dump trucks must not have yellow parts").

The specific setting studied involves an agent that:

Is initially unaware of the relevant structure concepts (e.g., specific truck types) and component part concepts.
Must acquire this knowledge piecemeal during deployment through user interactions.
Must adapt its behavior on the timescale of the current interaction to avoid violating these newly learned constraints.
Operates in a scenario where demonstrations alone are insufficient because multiple semantic rules could explain a correction, and the agent may not even know the concepts (e.g., "flat chassis center") referenced in the constraints.

Methodology
The authors propose a neurosymbolic architecture designed for online adaptation, separating dense perceptual grounding from explicit symbolic memory and planning. The system operates within a simulated toy-truck assembly domain.

Architecture Components:
- Vision Processing Module: Maps RGB images to structured symbolic representations, outputting candidate atomic part concepts with confidence scores and 3D geometric representations for join relations.
- Dialogue Management Module: Interprets teacher utterances using a semantic parser. It handles goal specification, clarification, and crucially, generic statements expressing semantic constraints (e.g., "All fenders of fire trucks are red"). These are translated into First-Order Logic (FOL) and then into Answer Set Programming (ASP) clauses.
- Symbolic Reasoning Module: Utilizes ASP (specifically the Clingo solver) for two subproblems:
  1. Goal Selection: Selects the best set of recognized objects to instantiate a goal structure, scoring candidates based on visual confidence and adherence to known constraints ( $S_\gamma$ ).
  2. Join-Sequence Planning: Plans a collision-free order of joins, integrating motion planners to check feasibility.
- Long-term Memory: Stores a Visual Exemplar Base ( $X_B$ ) for part classification, a Symbolic Knowledge Base ($KB$) containing domain theory and constraints, and a Lexicon mapping internal concept indices to natural language labels.
- Actuation Module: Executes primitive actions via an oracle controller.
Interaction Protocol:
The agent engages in a controlled dialogue with a teacher. If the agent lacks a concept, the teacher provides definitions or exemplars. If the agent makes a geometrically feasible but semantically invalid move, the teacher provides a generic constraint (e.g., "Coloured parts of a dump truck must not be yellow") rather than just a specific correction. The agent translates these into ASP rules to update its planning constraints immediately.
Experimental Strategies:
The study compares four strategies to isolate the value of different information types:
1. NoLabels+CaseMemory: No shared vocabulary; learns only from specific correction cases.
2. LabelsOnly: Uses shared part labels but no semantic constraints.
3. Labels+CaseMemory: Uses labels and specific correction cases ("Use X instead of Y").
4. Labels+TeacherRules (Full Approach): Uses labels and generic semantic constraints provided by the teacher.

Key Contributions

Neurosymbolic Framework for Online Semantic Adaptation: A system that decouples visual perception from symbolic reasoning, allowing the agent to update its knowledge base (lexicon, exemplars, and constraints) and replan immediately upon receiving new semantic information during task execution.
Formalization of Semantic Constraints in Assembly: The definition of assembly goals not just as topological graphs ( $G_\gamma$ ) but as structures constrained by quantified FOL rules ( $S_\gamma$ ) regarding part attributes and types.
Empirical Validation of Generic Constraints: The demonstration that communicating semantic constraints via natural language generics is more effective for online learning than relying solely on demonstrations, part labels, or specific correction cases.

Results
Experiments were conducted on 30 datasets, each containing 40 online assembly episodes with increasing complexity (introduction of new subtypes and distractors).

Cumulative Regret: The Labels+TeacherRules strategy consistently achieved the lowest cumulative regret (accumulated errors).
- NoLabels+CaseMemory performed significantly worse, highlighting the difficulty of learning without a shared vocabulary.
- LabelsOnly and Labels+CaseMemory showed similar performance, suggesting that storing specific correction cases is not an effective substitute for reusable semantic constraints in this domain.
Visual Grounding (F1 Score): All label-aware strategies achieved comparable visual grounding performance. The superior performance of Labels+TeacherRules was not due to better visual classification but rather to the symbolic planner's ability to avoid semantically invalid assemblies using the acquired constraints.
Data Efficiency: Communicating semantic constraints through natural language generics yielded more data-efficient online adaptation than relying only on task demonstrations or naming parts.

Significance and Claims
The paper claims to illustrate a specific form of neurosymbolic adaptation where explicit symbolic knowledge acquired during interaction immediately reshapes action selection over uncertain perceptual groundings.

The authors emphasize that their approach targets the online adaptation problem where domain knowledge is introduced piecemeal. By separating perception from reasoning, the system can update its lexicon and knowledge base without requiring retraining of the visual feature backbone. The study concludes that while visual grounding is necessary, the ability to acquire and exploit reusable semantic constraints via natural language is the critical factor for efficient learning in semantic assembly tasks.

The work is modestly scoped to a controlled simulated domain with a modular, primarily perception-to-reasoning coupling. The authors note that future work is needed to address robust perception, cross-domain transfer, and tighter symbolic-to-neural feedback for representation-level revision.

The Story of the Talking Robot Builder

The Four Teaching Styles

The Big Discovery: Rules Beat Examples

Why This Matters (and What It Isn't)

Technical Summary: Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints

More like this