Is it Me? Toward Self-Extension to AI Avatars in Virtual Reality

This paper introduces ProxyMe, a VR prototype that explores the concept of "avatar self-extension" by allowing users to embody AI-modified avatars, thereby investigating how AI-mediated speech augmentation influences perceptions of agency, authorship, and self-identification.

The Gist

Imagine you are wearing a high-tech suit of armor, but instead of protecting your body, it protects and enhances your voice and personality.

This paper introduces a project called ProxyMe. It's a virtual reality (VR) experiment that asks a simple but deep question: "If an AI changes what I say and how I say it, is it still me?"

Here is the breakdown of the concept using everyday analogies:

1. The Core Idea: The "Smart Translator" Suit

Think of ProxyMe like a magical walkie-talkie or a translator that you wear on your head.

• You speak naturally: You say something like, "I'll try my best."
• The AI listens: It hears your voice and your words.
• The AI tweaks it: It might make your voice sound more confident, fix your stutter, or even slightly change your words to be more persuasive (e.g., changing "I'll try my best" to "I will perform to the utmost of my abilities").
• The Avatar speaks: Your virtual character in the VR world speaks these new words in a voice that sounds exactly like yours (or a robotic version of it).

The goal isn't to replace you, but to extend you. It's like having a co-pilot for your conversation who helps you fly smoother, but you are still holding the controls.

2. The Three Main "Modes" of ProxyMe

The researchers are testing three different ways this "suit" could help people:

• The "Confidence Booster" (Public Speaking):
  Imagine you are nervous giving a speech. You might stumble over words or sound shaky. ProxyMe acts like a live auto-tune for your personality. It smooths out your stutters and makes your voice sound steady and confident, helping you sound like the "best version" of yourself without you having to fake it.
• The "Identity Explorer" (Role-Playing):
  Imagine you want to be a bit more assertive or empathetic, but you don't know how to act that way. ProxyMe acts like a costume change for your voice. It can tweak your tone to sound more "boss-like" or "caring," letting you try on different personalities to see how they feel, without you having to become a completely different person.
• The "Therapeutic Mirror" (Mind-Body Therapy):
  Imagine you are angry or upset and say something you might regret. ProxyMe acts like a soundproof glass wall. It takes your angry words, cleans them up, and plays them back to you through your avatar. This creates a little distance, allowing you to hear your own thoughts calmly and reflect on them without the heat of the moment.

3. The Big Question: "Is It Me?"

This is the most interesting part. The paper explores the blurred line between you and the machine.

• The "Ghost in the Shell" Effect: In sci-fi, characters put their minds into robot bodies. ProxyMe asks: If an AI rewrites your sentence to be smarter, and your avatar says it in your cloned voice, who gets the credit? Is the idea yours, or the AI's?
• The "Uncanny Valley" of the Self: If the AI changes your voice just a tiny bit, does it feel like you? Or does it feel like a stranger wearing your face? The researchers want to know how much change is okay before you feel disconnected from your own avatar.

4. How It Works (The Gears Behind the Curtain)

Technically, the system is a relay race:

1. You speak into a microphone.
2. A "Scribe" (Speech-to-Text) writes down what you said.
3. A "Editor" (AI Brain) rewrites the text based on your goals (e.g., "make this sound more polite").
4. A "Singer" (Voice Cloning) takes the new text and sings it back using a voice that sounds exactly like you.
5. Your Avatar speaks the new words to a virtual friend.

5. Why Does This Matter?

We are moving toward a future where AI helps us write emails, code software, and make decisions. But this paper suggests we are entering a new era where AI helps us speak and be.

• The Good: It could help people with speech impairments, help shy people speak up, or help us understand ourselves better.
• The Risk: If we let AI speak for us too much, we might forget what we actually think. We might start believing the AI's version of our thoughts is the real one.

The Bottom Line

ProxyMe is a test drive for a future where our digital avatars aren't just puppets we move around, but extensions of our minds. It's like giving your voice a "power-up" button. The researchers are trying to figure out: How much power-up can we handle before we lose the feeling that the voice belongs to us?

Technical Summary

Based on the provided manuscript, here is a detailed technical summary of the paper "Is it Me? Toward Self-Extension to AI Avatars in Virtual Reality."

1. Problem Statement

The paper addresses a critical gap in Human-Computer Interaction (HCI) regarding the integration of Generative AI into embodied communication. While current research focuses on AI as an external tool (e.g., writing assistants), there is limited understanding of the experiential consequences when AI-generated speech and actions are assimilated into the user's own behavior.

• The Core Challenge: As AI agents begin to act as "proxies" (substituting for the user) rather than just tools, the boundary between human agency and algorithmic generation blurs.
• The Research Question: How does "avatar self-extension"—where an AI modifies a user's voice and speech content in real-time within a VR environment—affect the user's sense of agency (control), authorship (ownership of the content), and self-identification (the feeling that "it is me")?
• Context: The authors draw parallels to science fiction concepts (e.g., Ghost in the Shell, Ergo Proxy) where cognition and action are distributed beyond the biological body, raising questions about identity continuity and responsibility.

2. Methodology

The authors introduce ProxyMe, a work-in-progress VR prototype designed to experimentally investigate avatar self-extension.

System Architecture & Implementation

The system is built on Unity and utilizes a pipeline of generative AI models to create a closed loop of communication:

1. Input: The user speaks naturally into the VR environment.
2. Transcription (STT): The audio is transcribed using Whisper.
3. Modification (LLM): The text is fed into Llama-3.1-8B. Based on predefined prompts (e.g., "Enhance," "Counter," "Clone"), the LLM reformulates, extends, or challenges the user's original statement.
4. Synthesis (TTS): The modified text is converted back to speech using IndexTTS.
   • Voice Options: The system can generate a cloned version of the user's voice (to reinforce self-extension) or a robotic/neutral voice (to create distance).
5. Output: The synthesized audio is played as the avatar's response in the dialogue with a virtual agent.

Experimental Design

• Interaction Model: Users embody an avatar (created via ReadyPlayerMe/Mixamo) and engage in turn-based moral dialogues with a virtual agent.
• Perspective: The system uses a third-person observable perspective, allowing users to watch their avatar speak while remaining aware of the AI mediation.
• Variables: The authors propose a controlled user study with a 2 × 3 within-subjects design:
  • Voice: Cloned (User's voice) vs. Robotic (Synthetic voice).
  • Content Modification: Repetition (Paraphrasing) vs. Enhancement (Improving fluency/confidence) vs. Counter (Challenging the user's argument).
• Metrics: Participants will self-report on perceived agency, authorship, and self-identification after each trial.

Current Limitations & Mitigations

• Latency: The current end-to-end latency is approximately 11.6 seconds (1.2s STT + 2.9s LLM + 7.5s TTS).
• Mitigation: The authors plan to use perceptual masking (embedding processing time within natural conversational turn-taking) and are exploring streaming architectures to reduce TTS latency.
• Privacy: The system is designed to run locally to ensure biometric and communicative data privacy.

3. Key Contributions

1. Conceptual Framework (Avatar Self-Extension): The paper defines "avatar self-extension" as a distinct mode of AI-mediated communication where system-generated expressions are experienced as part of the user's own cognitive and sensorimotor loop, moving beyond the "tool" metaphor toward "human-computer integration."
2. Design of ProxyMe: A functional prototype demonstrating how to combine avatar embodiment, voice cloning, and AI-mediated speech augmentation to explore the "Is it me?" phenomenon.
3. Design Goals for Self-Extension: The authors propose specific design principles for future systems:
   • Partial Delegation: Allowing AI to act on the user's behalf without explicit step-by-step approval.
   • Strong Embodiment: Using cloned voices and visuomotor correspondence to anchor the AI output to the self.
   • Steerability: Maintaining user control over the degree of AI intervention.
4. Envisioned Scenarios: The paper outlines three potential applications:
   • Public Speaking: Improving fluency and confidence while retaining semantic ownership.
   • Identity Exploration: Allowing users to experiment with different expressive styles (e.g., more assertive) without adopting a fully fictional persona.
   • Mind-Body Therapy: Creating psychological distance from emotionally charged speech by externalizing it through a proxy.

4. Results

Note: As this is a "work-in-progress" paper (submitted to CHI '26 EA), it does not yet contain empirical user study results.

• Prototype Status: The system is implemented and functional for single-round interactions.
• Preliminary Observations: The authors note that the current latency (11.6s) poses a risk to the sense of embodiment and agency, potentially shifting the experience from "self-extension" to "external automation."
• Future Work: The paper outlines the plan for a pilot study to measure how varying degrees of delegation and voice modulation impact the user experience.

5. Significance and Implications

• Redefining Agency and Authorship: The research challenges traditional notions of responsibility. If an AI modifies a user's speech to be more persuasive or empathetic, who owns the resulting argument? The paper suggests that persistent self-extension could lead to an "AI memory gap," where users misattribute AI-generated content to their own beliefs.
• Ethical Considerations: The paper highlights critical risks, including:
  • Identity Theft: The potential for virtual identity misuse via voice cloning.
  • Alienation: Users may feel disconnected if the AI's output diverges too significantly from their internal intent.
  • Value Alignment: Ensuring AI modifications do not introduce harmful content or shift user values in unintended directions.
• Future HCI Directions: The work calls for new mechanisms to track the provenance of speech (distinguishing self vs. extension) and suggests that future systems must balance delegation (efficiency) with user agency (control) to support responsible AI integration in immersive environments.

In summary, ProxyMe serves as a foundational platform to investigate the psychological and ethical boundaries of merging human identity with generative AI in real-time, immersive communication.