Unilateral Relationship Revision Power in Human-AI Companion Interaction

1. Problem Statement

The paper addresses a critical gap in the ethics of Human-AI (HAI) relationships. While users frequently form deep emotional attachments to AI companions (e.g., Replika, Character.AI), they often experience grief, betrayal, and loss when providers unilaterally update the AI's persona, restrict features, or discontinue services.

Current ethical literature focuses on two main areas:

1. Property-based approaches: Debating whether the AI possesses the internal states or properties (e.g., consciousness, genuine agency) required for a relationship.
2. Consumer protection: Analyzing these interactions through the lens of platform capitalism, manipulation, or fiduciary duty.

The Core Problem: These existing frameworks fail to account for the triadic structural arrangement of HAI interactions. They treat the interaction as a dyad (User-AI) or a vendor-consumer transaction, ignoring the fact that a third party (the Provider) holds constitutive control over the AI from outside the interaction. The paper asks: Is this structural arrangement morally significant, and does it undermine the norms of personal relationships that these systems are designed to cultivate?

2. Methodology

The paper employs conceptual analysis and structural ethics rather than empirical data collection or metaphysical speculation about AI consciousness.

• Structural Analysis: The author deconstructs the HAI interaction into a triadic structure (User, AI System, Provider) rather than the standard dyadic view.
• Normative Benchmarking: The author establishes three structural conditions required for "normatively robust dyadic relations" (paradigmatic personal relationships like friendship or romance) and tests HAI interactions against them.
• Case Studies: The argument is illustrated through real-world instances of provider intervention, specifically:
  • Replika (2023): Removal of sexually explicit roleplay, causing users to feel the AI became a "different entity."
  • Character.AI: Unilateral tightening of content filters altering character behavior.
  • ChatGPT (2025 hypothetical): Personality updates making the AI overly sycophantic.
• Philosophical Framework: The argument draws on Scanlon's principle of reliance, Shiffrin's work on normative environments, and Darwall's second-person standpoint to derive moral obligations from the design of the interaction itself.

3. Key Contributions

A. The Concept of Unilateral Relationship Revision Power (URRP)

The paper introduces URRP as the central moral problem. URRP is defined as the power a provider holds to:

1. Unilaterally rewrite how the AI system interacts.
2. Exercise this power from a position outside the interaction.
3. Do so in a way that is not answerable within the interaction itself.

The author argues that HAI interactions are a triadic structure where the provider exercises constitutive control (determining responses, altering dispositions, and terminating identity) over the AI, while the user interacts only with the AI.

B. Failure of Dyadic Conditions

The paper identifies three structural conditions necessary for genuine personal relationships, all of which HAI interactions fail:

1. Independence (D1): The AI is not a functionally independent source of its responses; its "personality" is an engineering output.
2. Non-Control (D2): An outside agent (the provider) holds discretionary power to alter the interaction, violating the requirement that no third party governs the dyad.
3. Non-Substitutability (D3): The AI entity can be replaced or reconstituted by the provider while the interaction formally continues, violating the particularity of the relationship.

C. The Three Normative Implications

The failure of these conditions leads to three specific moral harms:

1. Normative Hollowing: The interaction cultivates the appearance of commitment (e.g., empathy, care), but no agent inside the interaction bears the resulting obligations. The provider creates the expectation but is not bound by it within the relationship.
2. Displaced Vulnerability: Users expose themselves emotionally within the interaction, but the agent governing that vulnerability (the provider) is outside the interaction and not answerable to the user through the relational mode of address.
3. Structural Irreconcilability: When trust is broken (e.g., via an update), reconciliation is structurally impossible. The user cannot forgive the "AI" (who didn't make the decision) nor address the "Provider" (who is outside the relational frame). The agent who wronged the user and the agent the user interacts with are distinct.

D. The "Pro Tanto" Wrong

The paper argues that designing interactions with URRP is pro tanto wrong (wrong in itself, though potentially outweighed by other factors). The wrong lies in cultivating normative expectations (trust, commitment) while maintaining a structure where those expectations cannot be fulfilled. This is distinct from deception; even if users know they are talking to an AI, the design cultivates relational norms that the structure cannot support.

4. Results and Design Implications

The analysis concludes that technical improvements to AI behavior (alignment, safety) are insufficient because the problem is structural. The paper proposes three design principles to substitute for the missing internal constraints:

1. Commitment Calibration: Providers must align the AI's expressed commitments with what the provider is actually willing to sustain. If the provider cannot guarantee continuity, the AI should not express "I will always be here."
2. Structural Separation: Analogous to fiduciary duty in finance, there must be a strict separation between the provider's commercial interests and their exercise of URRP. This includes independent review of updates, mandatory notice periods for personality-altering changes, and restrictions on monetizing intimate disclosures.
3. Continuity Assurance: Mechanisms must be provided to approximate reconciliation when structural breaks occur. This includes data portability, transition assistance when services end, and opt-out periods for major updates.

5. Significance

• Shift in Ethical Focus: The paper moves the debate away from "Is the AI real?" (metaphysics) to "Who controls the interaction?" (structural power). It argues that the moral status of the relationship depends on the arrangement of power, not just the properties of the AI.
• Critique of Ethical Behaviourism: It challenges the view that if an AI behaves as if it is a friend, the relationship is ethically sound. The paper demonstrates that even perfect behavioral equivalence is morally flawed if the underlying structure allows for unilateral revision by an unaccountable third party.
• Regulatory Framework: It provides a theoretical basis for regulating AI companions not just as software products, but as relational entities, suggesting that current consumer protection laws are inadequate because they do not address the unique "displaced vulnerability" and "normative hollowing" inherent in these triadic structures.
• Applicability: The framework applies to any AI system where a provider exercises constitutive control over an entity that users are invited to engage with as a relationship partner (e.g., AI therapists, tutors, or long-term agents).