🤖 AI

Fix the Mind, Not the Move: Interpretable AI Assistance via Knowledge-Gap Localization

The paper introduces SENSEI, an interpretable AI framework that improves long-term human-AI collaboration by localizing and correcting underlying user misconceptions through targeted suggestions rather than merely fixing immediate actions, achieving high success rates in zero-shot generalization across diverse tasks.

Original authors: Ayano Hiranaka, Ya-Chuan Hsu, Stefanos Nikolaidis, Erdem Bıyık, Daniel Seita

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ayano Hiranaka, Ya-Chuan Hsu, Stefanos Nikolaidis, Erdem Bıyık, Daniel Seita

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Fixing the Recipe, Not Just the Cooking

Imagine you are teaching a friend to cook a complex meal. Your friend keeps making the same mistake: they try to boil an egg by putting it directly into a microwave.

The Old Way (Fixing the Move): A standard AI assistant might just yell, "Stop! Don't do that!" or physically nudge the microwave door shut. This stops the immediate disaster, but your friend still thinks, "Oh, microwaves are fine for eggs," and they will likely try it again tomorrow with a potato. They fixed the action, but not the thinking.
The New Way (SENSEI): The system introduced in this paper, called SENSEI, acts like a master chef who realizes, "My student doesn't know that sealed shells explode in microwaves." Instead of just stopping the hand, SENSEI explains the rule: "Sealed, liquid-filled shells build pressure and explode." Now, the student learns a new rule. They won't just avoid microwaving eggs; they won't microwave sealed potatoes or jars either. They have fixed the mind.

What is SENSEI?

SENSEI is a framework designed to help AI assistants understand why a human is making mistakes in long, complex tasks (like cooking, driving, or planning a rover mission) and then teach them the correct rule so they don't make those mistakes again.

It operates on a simple principle: People usually act logically based on what they believe to be true. If they do something "wrong," it's often because their internal "rulebook" for the world is slightly broken.

How Does It Work? (The Detective and the Editor)

SENSEI works in two main steps, acting like a detective followed by a editor:

The Detective (Localization):
Imagine the expert's knowledge is a library of 100 rulebooks (e.g., "How to hold a tomato," "How to open a door," "How to clean a pan"). The student is following a different set of rules.
SENSEI watches the student try to do a task. It compares their actions to an expert's actions. Instead of guessing randomly, it looks at the specific "rulebooks" (knowledge components) to find exactly which one is wrong.
- Analogy: It's like a mechanic listening to a car engine. Instead of saying "The car is broken," it pinpoints exactly which part is making the noise: "Ah, it's the spark plug, not the tires."
The Editor (Correction):
Once SENSEI finds the broken rule, it doesn't just delete it; it rewrites it to match the expert's version. It then translates this technical fix into simple, natural language advice for the human.
- Analogy: If the student thought "Doors only open by clicking," SENSEI edits that rule to "Doors open by clicking OR pressing 'A'." It then tells the student: "Hey, try pressing 'A' on the door."

The Magic Trick: Learning from One Mistake to Fix Many

One of the paper's coolest findings is Zero-Shot Compositional Generalization.

Imagine you train a student to recognize that "Salt looks like Sugar" is a mistake. You only show them that one specific error.

Old AI: If you then show them a new error, like "Tomatoes look like Onions," the old AI might get confused because it hasn't seen that exact combination before.
SENSEI: Because it understands the structure of the mistake (confusing object types), it can instantly recognize and fix new combinations of mistakes it has never seen before. It's like teaching someone the concept of "confusion" rather than just memorizing a list of confused items.

The Results: Does It Actually Work?

The researchers tested SENSEI in three different "worlds":

Breakfast: A cooking task involving eggs, milk, and cleaning.
Overcooked: A chaotic kitchen game where you have to make soup and serve it.
Rover: A space robot mission to collect rocks and send photos.

The Findings:

Accuracy: SENSEI successfully identified and corrected about 90% of the human misconceptions in a user study.
Efficiency: It didn't just guess; it found the right rule to fix. Other methods either guessed too many wrong rules (causing confusion) or missed the real problems entirely.
Human Feedback: In a study with 20 real people, users found the advice helpful. Even though the system sometimes gave advice they already knew (false alarms), the users still rated it highly because the correct advice was so valuable.

The Bottom Line

SENSEI changes the goal of AI assistance. Instead of being a "traffic cop" that just stops you from making a bad move, it acts as a "tutor" that updates your internal map of how the world works. By fixing the mind (the underlying knowledge), it ensures the move (the action) is correct now and in the future.

Where is this used?
According to the paper, this is currently tested in simulated planning environments (like cooking games and robot missions). The authors emphasize that this is a step toward making AI assistants that can truly teach humans, rather than just controlling them.

Technical Summary: Fix the Mind, Not the Move: Interpretable AI Assistance via Knowledge-Gap Localization

Problem Statement
Current AI assistants in human–AI collaboration often rely on behavioral feedback (e.g., steering, alerts, or overrides) to correct suboptimal human actions in real-time. While effective for immediate error mitigation, these "move-fixing" interventions fail to address the underlying misconceptions that cause repeated systematic errors. The paper argues that robust mastery in long-horizon decision-making tasks requires correcting the human's internal task knowledge, not just their immediate trajectory. The core challenge is to infer a human's specific knowledge gaps (misconceptions) from observed behavior traces and provide targeted, interpretable guidance that aligns the human's future behavior with an expert's, even on new task instances.

Methodology: SENSEI
The authors propose SENSEI (Structured Extraction and Neural Synthesis of Errors for Intervention), a framework that operates over a structured knowledge representation (specifically Planning Domain Definition Language, or PDDL) to localize and correct knowledge gaps. The system follows a two-stage inference pipeline:

Knowledge Gap Localization: Given expert knowledge ( $K_E$ ), expert behavior ( $\tau_E$ ), and student behavior ( $\tau_S$ ), a Localization Network ( $f_{loc}$ ) predicts which specific symbolic knowledge components (e.g., object types, action preconditions, or effects) are misaligned between the student and the expert. To handle multiple overlapping misconceptions, the authors employ a Mixup Generalization Loss, training the network on synthetic embeddings created by interpolating between single-misconception behaviors. This enables zero-shot compositional generalization to multi-misconception scenarios despite training only on single-misconception data.
Latent Knowledge Editing and Decoding: Once a gap is localized, a Latent Knowledge Editor ( $f_{edit}$ ) predicts a latent edit vector to transform the expert's knowledge component embedding into the student's variant. A Student Knowledge Decoder ( $f_{dec}$ ), based on a CodeT5+ architecture, then translates this edited embedding back into natural language or raw PDDL text, describing the student's specific misconception.

The framework is trained to minimize three objectives: accurate gap localization, accurate correction of true gaps, and minimal/no edits on non-gap components (to avoid hallucinating errors).

Key Contributions

Formulation of Knowledge-Aware Assistance: The paper formalizes AI assistance as a knowledge correction problem, aiming to align student knowledge ( $K_S$ ) with expert knowledge ( $K_E$ ) to minimize future regret, rather than simply optimizing immediate trajectory alignment.
Structured Diagnose-then-Edit Framework: SENSEI decouples the "where to edit" (localization) from "how to edit" (synthesis). This structural constraint ensures corrections are minimal, interpretable, and grounded in specific symbolic knowledge components, reducing spurious updates.
Zero-Shot Compositional Generalization: The system demonstrates the ability to disentangle and correct multiple overlapping misconceptions in a zero-shot manner, despite being trained exclusively on data containing single misconceptions.
Empirical Validation: The authors validate the approach across three complex planning domains (Breakfast, Overcooked, and Rover) and a user study with 20 participants.

Results

Simulation Experiments: Across three domains, SENSEI achieved the best balance between correcting true gaps and avoiding erroneous corrections. For instance, in the Overcooked domain, SENSEI achieved a System F1 score of 0.726, significantly outperforming monolithic end-to-end models (0.028) and off-the-shelf LLMs (0.005–0.076). The localization module successfully identified the correct components (LocRecall $\approx$ 0.83 in Overcooked) while maintaining high precision.
Ablation Studies: Removing the Mixup loss caused a catastrophic drop in system recall (e.g., from 0.762 to 0.221 in Breakfast), confirming its necessity for generalizing to multi-misconception cases. Removing the expert reference during training led to higher precision but lower recall, suggesting the expert anchor is crucial for sensitivity to deviations.
User Study: In a modified Overcooked task, SENSEI successfully identified and corrected real human misconceptions induced by withheld instructions. The system achieved a System Recall of 0.895 and improved the alignment between user and expert behavior (IoU) from 0.736 to 0.844. While users received some unhelpful tips (false positives), they rated the system's overall helpfulness highly (4.27/5), indicating that knowledge-aware guidance is valuable even with occasional errors.

Significance and Claims
The paper claims that SENSEI represents a shift from "fixing the move" (shared autonomy) to "fixing the mind" (knowledge correction). By localizing symbolic knowledge gaps and synthesizing minimal, interpretable corrections, the framework provides a foundation for AI assistance that supports long-term learning and generalization. The authors emphasize that their approach moves beyond opaque behavior steering toward transparent, logic-based diagnosis.

However, the paper maintains a modest stance regarding its scope and limitations. It acknowledges that the reliance on expert reference trajectories and predefined PDDL domains implies a normative definition of "optimality," which may penalize valid alternative strategies in open-ended settings. Furthermore, the authors note that diagnostic systems are prone to erroneous corrections (hallucinating misconceptions), which can lead to user frustration. Consequently, they position SENSEI as a supportive tool rather than an autonomous evaluator, particularly in high-stakes environments where diagnostic accuracy is critical. The work does not claim to solve all human learning problems but rather offers a structured method for aligning human task knowledge with expert models in long-horizon decision-making.