Cognitive Digital Twins: Ethical Risks and Governance for AI Systems That Model the Mind

Technical Summary: Cognitive Digital Twins: Ethical Risks and Governance for AI Systems That Model the Mind

Problem Statement
As AI systems evolve from mere assistants to entities that model, simulate, and act on behalf of individuals, a new class of technology emerges: Cognitive Digital Twins (CDTs). Unlike traditional digital twins that model physical bodies or processes, CDTs are dynamic computational representations of a specific person's cognition (states, dispositions, processes). The core problem identified is that existing governance frameworks—designed for data processing, automated decision-making, or autonomous agents—fail to address the unique risks of CDTs. These risks occur not merely at the point of a final decision or external action, but at the earlier stage where a model's claim about a person's cognition becomes trusted and operationalized. Current frameworks do not adequately regulate the representation, simulation, and delegation of cognition itself, leading to potential harms such as misrepresentation, epistemic authority shifts, and the creation of "shadow twins" without consent.

Methodology and Analytical Framework
The paper employs a conceptual and normative analysis to define CDTs and evaluate their governance requirements. It does not present empirical experimental results but rather constructs a theoretical framework to analyze the intersection of cognitive science, AI safety, and ethics.

The authors introduce the 5A Framework to characterize CDT power and identify governance duties:

1. Authority: The scope of actions the CDT can authorize or perform on behalf of the person (e.g., suggesting vs. committing to decisions).
2. Autonomy: The degree to which the CDT can initiate actions or simulations proactively without explicit instruction.
3. Access and Control: The rights of the represented person to inspect, correct, configure, export, or retire the CDT and its data.
4. Accountability: Mechanisms to trace harms, identify responsible parties (developers, deployers, users), and provide redress.
5. Availability: Ensuring equitable access to beneficial CDT capabilities while preventing the concentration of proxy power among powerful actors.

Using this framework, the authors analyze two hypothetical near-future scenarios: a "simulated patient" in mental health care and a "productivity twin" in the workplace. These cases illustrate how CDTs can be used for simulation (predicting responses to interventions) and proxy action (communicating or deciding on behalf of the user).

Key Contributions
The paper makes four primary contributions:

1. Definition and Distinction: The authors define CDTs as dynamic, person-specific computational representations of cognition updated via behavioral, contextual, or physiological data. They distinguish CDTs from adjacent systems (e.g., recommender systems, digital phenotyping, companion chatbots) based on the operationalization of person-directed cognitive claims. A system becomes a CDT when it models specific cognitive states (memory, belief, vulnerability) to simulate, classify, or act as a proxy, rather than merely optimizing for user preference or engagement.

2. Identification of CDT-Specific Risks: The paper identifies a distinct set of risks arising from the operationalization of cognitive models:

   • Misrepresentation and Robustness: Errors in modeling reasons, values, or identity can become self-reinforcing (e.g., a model predicting disengagement leading to clinician behavior that causes disengagement).
   • Epistemic Authority Shifts: The locus of self-knowledge may shift from the person to the proxy, where the model's predictions override lived experience or self-report, potentially creating self-fulfilling prophecies.
   • Shadow Twins: The construction of individualized cognitive models without meaningful consent, often for dual-use purposes (e.g., an insurer simulating non-adherence or a platform simulating vulnerability for manipulation).
   • Simulation Without Participation: The substitution of actual human participation with simulated responses for decision-making, potentially bypassing consent and democratic consultation.
   • Delegated Proxy Action and Source Gaps: The blurring of lines between actions taken by a person, for a person, or about a person, creating "source gaps" where recipients cannot distinguish between personal endorsement and proxy output.

3. Governance Gap Analysis: The authors analyze existing frameworks (GDPR, EU AI Act, NIST AI RMF) and identify specific gaps:

   • Data Rights vs. Cognitive Control: Consent to data processing does not equate to consent to cognitive modeling or simulation.
   • Timing of Triggers: Current risk categories often trigger only after a final automated decision, missing harms that occur during the simulation or interpretation phase.
   • Agent vs. Simulation: Governance for autonomous agents focuses on external action, missing the influence of non-agentic simulations that shape institutional behavior.
   • Cognitive-Claim Discipline: Existing risk management lacks specific protocols for validating the cognitive constructs a system claims to model.

4. Proposed Governance Requirements: The paper proposes a set of concrete requirements for high-risk CDT deployments (summarized in Table 2), including:

   • Layered Consent: Distinct permissions for data collection, modeling, simulation, and proxy action.
   • Purpose Binding: Technical and legal mechanisms to prevent silent repurposing of models.
   • Robustness Evaluation: Testing for construct validity, calibration, temporal drift, and subgroup performance.
   • Traceability: Audit logs for data provenance, model versions, and institutional uptake of CDT outputs.
   • Proxy-Action Labels: Disclosure when communications or decisions are materially shaped by a CDT.
   • Contestation and Retirement: Mechanisms for users to dispute inferences and retire stale or unsafe models.
   • Independent Review: Risk-proportionate review for institutional simulations and high-stakes proxy actions.

Results and Claims
The paper does not present experimental data but offers a structured argument that CDTs represent a fundamental shift in AI governance. The "result" is the identification of a governance gap where cognition is treated as an object of representation and delegation without specific regulatory safeguards. The authors argue that CDTs require governance not only because they can act for people, but because they become infrastructures through which cognition is represented, simulated, classified, and operationalized.

Significance
The paper claims that the significance of CDTs lies in their ability to transform how people are known and interacted with by institutions. The ethical stakes are high because CDTs can fragment social agency, create "moral crumple zones" where humans absorb blame for automated failures, and enable manipulation through deep cognitive modeling. The authors assert that effective governance must move beyond regulating data and decisions to regulating the cognitive representation itself, ensuring that the delegation of cognitive agency does not undermine human autonomy, dignity, or democratic participation. The work serves as a foundational definition and risk assessment for a technology class that is rapidly converging from adjacent fields like digital phenotyping, neuroprosthetics, and generative agents.