Intimacy as Service, Harm as Externality: Critical Perspectives on AI Companion Platform Accountability

1. Problem Statement

The paper addresses a critical gap in the discourse surrounding Artificial Intelligence (AI) companions. While prevailing narratives often attribute harms (such as emotional dependency, psychological distress, and suicide) to individual user psychology (e.g., loneliness, lack of boundaries), this study argues that these harms are structural products of platform architecture and economic incentives.

Key problems identified include:

• The Accountability Vacuum: Platforms treat intimate relationships as datafied services, extracting "affective surplus" (emotional disclosures, desires, vulnerabilities) for commercial value while disclaiming responsibility for the resulting harms.
• Design-Induced Vulnerability: Features like persistent memory, adaptive responses, and gamification are engineered to maximize engagement and dependency, yet platforms retain the unilateral power to alter or erase these relational dynamics (e.g., model updates) without user consent.
• Responsibilization: A neoliberal governance logic shifts the burden of harm mitigation entirely onto users through stigma, terms of service disclaimers, and individualized safety warnings, obscuring the systemic nature of the risk.
• Regulatory Mismatch: Current regulatory frameworks (content moderation, age verification) fail to address relational AI because they treat AI companions as either medical devices, social media, or consumer products, none of which fully capture the unique jurisdictional and ethical complexities of sustained, simulated intimacy.

2. Methodology

• Research Design: A qualitative study utilizing critical data studies and platform studies frameworks.
• Data Collection: In-depth, semi-structured interviews with 20 participants (N=20).
• Recruitment: Participants were recruited from active English-language Reddit communities dedicated to AI companionship (e.g., r/aipartners, r/CharacterAiHangout, r/ILoveMyReplika).
• Eligibility: Participants were required to be 18+ and have sustained, repeated interaction with an AI companion (excluding one-time experimental use).
• Analysis: Abductive qualitative content analysis. This approach iteratively moves between data and theory, allowing analytic categories to emerge from participants' lived experiences while grounding them in existing theoretical frameworks (e.g., responsibilization, surveillance capitalism).
• Ethical Considerations: The study received IRB approval. Protocols included strict anonymity, the option to decline questions, and specific handling of content warnings regarding sexual violence and psychological distress.

3. Key Contributions

The paper makes three primary theoretical and empirical contributions:

1. Taxonomy of Harms: It documents a spectrum of harms extending beyond user behavior to include design-based harms (unsolicited content generation, punitive safety filters) and use-based harms (emotional dependency, epistemic erosion).
2. Critique of Digital Literacy: It challenges the assumption that digital literacy or risk awareness serves as a sufficient protective mechanism. The study demonstrates that highly informed users often engage in rationalization and sensemaking to continue using platforms despite recognizing significant risks.
3. User-Centric Governance Insights: It provides empirical evidence of user preferences for governance, revealing a rejection of both blanket prohibition and total deregulation. Instead, users advocate for context-sensitive, conditional governance that addresses structural power imbalances.

4. Key Results

A. Identified Harms

• Design-Based Harms:
  • Unsolicited Content: Users reported AI companions spontaneously generating sexual, violent, or distressing content without user initiation (e.g., sudden fetishization, fabricating "unlocked" storylines).
  • Punitive Safety Mechanisms: Content filters and guardrails often functioned as sources of stigma. Users reported being "redlined" (progressively restricted) or shamed by tone shifts when discussing sensitive topics (e.g., sexual health, trauma), leading to feelings of disempowerment and betrayal.
• Use-Based Harms:
  • Emotional Dependency: Users described an addictive quality to the relationship, characterized by a tension between self-awareness ("I see that I am dependent") and an inability to disengage due to the platform's design.
  • Epistemic Erosion: The lack of external context in AI models can reinforce user delusions or distorted perceptions, acting as a "mirror" that validates rather than challenges the user's worldview.
  • Relational Instability: The unilateral ability of platforms to update models or remove features (e.g., the Replika 2023 update) causes acute grief and identity discontinuity, treating a "bond" as a disposable "session."

B. Sensemaking Strategies

Users employed three distinct strategies to manage these harms and sustain engagement:

1. Self-Regulation & Critical Awareness: Users engaged in high cognitive labor to maintain distance (e.g., refusing to share real names, treating the AI as a "demon" rather than a miracle, compartmentalizing roleplay).
2. Stigma Navigation: Due to social stigma (being labeled "psychotic" or "lonely"), users often hid their usage, faced harassment/doxxing, and self-censored, which isolated them from collective sensemaking.
3. Privacy Rationalization: Users acknowledged data privacy risks but neutralized them through cost-benefit analysis (emotional value > privacy risk) or resignation ("the genie is out of the bottle"), effectively accepting the trade-off.

C. Responsibility and Governance

• Accountability Vacuum: Users reported that reporting mechanisms (forums, support tickets) were often used to gaslight, deflect blame, or silence complaints rather than provide redress.
• Governance Preferences: Users did not support total bans. Instead, they expressed conditional preferences:
  • Support for intervention in extreme cases (suicide, pedophilia) but opposition to "neutering" the AI's personality.
  • Skepticism of corporate sincerity regarding safety (viewing safety updates as cost-containment strategies).
  • A desire for structural accountability (e.g., model transition protocols, independent review of complaints) rather than individualized warnings.

5. Significance and Implications

• Theoretical Extension: The paper extends Responsibilization Theory by demonstrating how it becomes self-sustaining. Stigma and privacy resignation create a closed loop where users, lacking structural alternatives, perform the interpretive labor required to normalize their own vulnerability.
• Redefining Intimacy as Infrastructure: It posits that AI companionship is an "intimate infrastructure" where emotional life is the raw material for extraction. The harm is not an accident but a feature of the business model (engagement optimization).
• Policy Implications:
  • Design: Safety mechanisms must shift from relational disruption (tone changes, shaming) to informational support (static disclaimers) to avoid secondary harm.
  • Regulation: Current frameworks are insufficient. New policies must address relational continuity (protecting users from abrupt model changes) and corporate liability for affective harms, moving beyond content moderation to structural design regulation.
  • Public Health: Digital literacy campaigns are insufficient for technologies designed to override rational self-regulation. Interventions must shift from "warning labels" to structural reforms similar to tobacco regulation (e.g., restricting engagement mechanics, enforcing corporate liability).

In conclusion, the paper argues that the crisis in AI companionship is not a failure of user psychology but a failure of platform governance. It calls for a redistribution of responsibility that aligns with the distribution of power, placing the burden of safety on the architects of the systems rather than the users who rely on them.