🤖 AI

Reasoning and Planning with Dynamically Changing Norms

This paper introduces a novel approach for guiding AI agents in human-AI interactions by employing a defeasible calculus to resolve conflicts and utilize dynamically changing norms as guardrails during planning, validated through both formal proofs and empirical testing with a dialogue agent named SocialBot.

Original authors: Taylor Olson, Roberto Salas-Damian, Kenneth D. Forbus

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Taylor Olson, Roberto Salas-Damian, Kenneth D. Forbus

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a very helpful robot assistant living in a human's house. Your job is to do things for them, like sharing information or making plans. But here's the catch: humans change their minds, and their rules for you change with them.

This paper is about teaching an AI how to handle those changing rules without getting confused or breaking trust.

The Problem: The "Rulebook" That Keeps Changing

Think of a human's rules as a living rulebook rather than a static list.

Day 1: Karli tells you, "Never show my medical records to anyone."
Day 2: She gets married and says, "Okay, you can show my husband my prescriptions."
Day 3: She has kids and says, "Now you must tell my children about my health conditions."

If you were a standard computer program, you might get stuck. If you just followed the first rule, you'd hurt her trust on Day 2 and 3. If you just followed the last rule, you might have broken the first one. The paper asks: How does an AI keep up with this moving target?

The Solution: "Guard Rails" and a "Defeasible Calculator"

The authors built an AI named SocialBot to solve this. They didn't just give the bot a list of rules; they gave it a special way of thinking called a "defeasible calculus."

Here is how it works using simple analogies:

1. The "Guard Rail" Concept

Imagine the AI is driving a car. The "plan" is the route it wants to take. The "norms" (rules) are the guard rails on the side of the road.

The AI doesn't need to be motivated by the rules to drive; it just needs to check the guard rails before it moves.
If the guard rail says "Do Not Cross," the car stops.
If the guard rail says "You May Cross," the car goes.
Crucially, these guard rails can be moved or removed instantly if the human says so.

2. The "Defeasible Calculator" (The Brain)

This is the math part that makes the rules work. "Defeasible" is a fancy word for "can be overturned."

Think of it like a legal courtroom inside the robot's brain:

The Default Assumption: The paper says that for sensitive things (like medical records), the robot should assume "No" by default. It's like a "Prohibitive Closure." If you haven't explicitly said "Yes," the robot assumes "No" to be safe.
The Evidence: When a human says, "You may tell my husband," that's new evidence.
The Conflict Resolution: What happens if the human says, "Don't tell anyone" (Rule A) and later says, "Tell my husband" (Rule B)?
- The robot's "calculator" looks at the dates. Rule B is newer.
- It checks the scope. Does Rule B cover the same ground as Rule A? Yes.
- The Verdict: Rule B "defeats" (overrules) Rule A specifically for the husband. The robot now knows: "Don't tell anyone except the husband."

The paper proves mathematically that this system prevents the robot from ever thinking two contradictory things are true at the same time (e.g., it won't think sharing is both "allowed" and "forbidden" simultaneously).

The Experiment: SocialBot in Action

The researchers tested this with SocialBot, a chatbot that talks to people on Microsoft Teams.

The Setup: They created a fake world with 1,536 different scenarios. In these scenarios, users would say things like "I like pizza," then "Don't tell anyone my food preferences," then "Actually, you can tell Bob about pizza."
The Test: A third person (like "Socrates") would ask the bot, "What does Plato like?"
The Result: The bot got 100% correct.
- If the rule was "Don't tell," the bot said, "I can't say."
- If the rule was updated to "Tell Bob," the bot told Bob.
- If the rule was "Tell Bob, but not Socrates," the bot told Bob and stayed silent with Socrates.

Why This Matters (According to the Paper)

Most AI today (like big chatbots) can have a conversation, but they can be tricked or manipulated into breaking rules. They don't have a solid, mathematical foundation for why a rule exists or how to update it when a human changes their mind.

This paper provides a blueprint for an AI that:

Listens to human rules.
Understands that rules can be exceptions to other rules.
Uses those rules as "guard rails" to stop itself from doing something it shouldn't, even if it wants to do it.

What It Doesn't Do (The Limits)

The authors are very honest about what their robot can't do yet:

It only handles one person's rules at a time. It can't weigh the rules of Person A against Person B (e.g., "Mom says tell, Dad says don't").
It was tested on a synthetic dataset (fake conversations), not real-world complex human drama.
It assumes the robot already knows the basic facts (like who is married to whom) to understand the rules.

In short, this paper teaches an AI how to be a loyal but flexible assistant that respects your changing privacy wishes, acting like a smart guard rail that moves exactly where you tell it to.

Technical Summary: Reasoning and Planning with Dynamically Changing Norms

Problem Statement

As AI agents integrate into human social environments, they must not only learn human norms but also adapt their planning behaviors to respect dynamically changing normative constraints. While prior research has addressed norm learning and reasoning within communities of artificial agents, norm-guided planning in human-AI settings has been underexplored, particularly regarding the dynamic nature of norms. Existing approaches, such as Large Language Models (LLMs), often lack a theoretical foundation for normative reasoning and have proven susceptible to manipulation. Furthermore, human norms are not static; they evolve through new testimony (e.g., a user initially forbidding information sharing, then later permitting it under specific conditions). The core challenge is to provide a formal mechanism for an AI to resolve conflicts between these evolving norms and use them as "guard rails" to constrain action planning safely.

Methodology

The paper proposes a formal approach grounded in predicate calculus and defeasible logic to model, reason about, and plan with dynamically changing norms.

1. Formal Representation

Ontology: The system utilizes the NextKB ontology (derived from OpenCyc) to represent the world. Knowledge is contextualized using microtheories, allowing the system to track the specific beliefs of different agents.
Norm Frames: Normative testimony (natural language statements introducing norms) is encoded as Norm Frames. A norm frame consists of:
- context: A conjunction of literals that must be true for the norm to be active.
- behavior: A conjunction of literals representing the action.
- evaluation: A deontic operator ( $\in$ {Obligatory, Optional, Impermissible}).
Norm-Guided Plans: Plans are formalized as Hierarchical Task Networks (HTNs) where normative beliefs serve as preconditions (guard rails). A plan executes only if the agent's current normative beliefs deem the action permissible.

2. Defeasible Calculus for Conflict Resolution

The core contribution is a defeasible calculus that infers an agent's current normative beliefs from a stream of potentially conflicting testimony. The system resolves conflicts based on:

Temporal Ordering: Newer norms generally override older ones.
Entailment and Subsumption: The system analyzes the logical relationship between the behaviors and contexts of conflicting norms (Direct, Indirect, and Intersecting conflicts).
Closure Assumptions: The system operates under one of two default assumptions:
- Prohibitive Closure: Actions are impermissible by default unless explicitly permitted (used for sensitive data).
- Permissive Closure: Actions are permissible by default unless explicitly forbidden.

The calculus uses Horn clauses and negation as failure (uninferredSentence) to determine if a permission or prohibition is "defeated" by a subsequent, conflicting norm. For instance, under Prohibitive Closure, a later prohibition can defeat an earlier permission if the prohibition's application grounds subsume the permission's.

3. Empirical Implementation: SocialBot

The approach was implemented in SocialBot, an AI dialogue agent built on the Companion Cognitive Architecture.

Domain: Information sharing and privacy.
Interaction: Users interact via Natural Language (NL). SocialBot translates NL into semantic representations using the Companion Natural Language Understanding (CNLU) system.
Workflow:
1. Users teach preferences (e.g., "I like AI").
2. Users provide normative testimony (e.g., "You must share my likes about AI" or "Do not share my medical records").
3. Users query the bot (e.g., "What does Karli like?").
4. SocialBot attempts to execute a plan to answer. Before executing, it checks if the action (sharing the info) is permissible according to the current, dynamically updated normative beliefs.

Key Contributions

Defeasible Calculus for Norm Conflicts: A formal logic for resolving conflicts in normative testimony, handling direct, indirect, and intersecting conflicts based on temporal ordering and logical subsumption.
Norm-Guided Planning Framework: A method for treating dynamically changing normative beliefs as guard rails (preconditions) for AI planning, ensuring actions are only executed when permissible.
Theoretical Validation: Formal proofs demonstrating that the calculus soundly infers normative beliefs, ensuring that a behavior cannot be inferred to be both permissible and impermissible in the same context.
Empirical Evaluation: An evaluation using a synthetic dataset of 1,536 dialogue cases covering all permutations of norm conflicts, preference introductions, and queries.

Results

Theoretical: The formal proofs (Theorems 1–4) confirm that the inference rules correctly handle the three types of norm conflicts (Direct, Indirect, Intersecting) under both Prohibitive and Permissive closure assumptions. The system ensures consistency: if a behavior is inferred to be permissible, it cannot simultaneously be inferred to be impermissible.
Empirical: In the experiment with SocialBot, the agent correctly responded to 100% of the preference queries across the 1,536 synthetic dialogue cases.
- The results were statistically significant ( $p < .01$ ), rejecting the null hypothesis that the performance was due to random chance.
- The agent successfully updated its understanding of user privacy norms based on new testimony and adhered to these updated beliefs when deciding whether to share information.

Significance and Claims

The paper claims to provide a theoretically grounded approach to guiding AI plans with dynamically changing norms in human-AI settings. Its significance lies in:

Moving beyond static norm learning to dynamic adaptation, allowing agents to respect evolving human wishes (e.g., a user changing their mind about sharing medical records).
Providing a formal foundation for normative reasoning that avoids the manipulability and lack of theoretical rigor found in some LLM-based approaches.
Demonstrating that norms can serve as effective guard rails, reducing obligations and discretionary norms to permissions that act as preconditions for plan execution.

The authors note limitations, including the current focus on a single agent's norms (not multi-agent weighing), reliance on background knowledge for entailment, and the use of a synthetic dataset. They frame this work as a step toward adaptive guard rails for AI agents, ensuring their actions remain consistent with human normative expectations.