CompanionCast: Toward Social Collaboration with Multi-Agent Systems in Shared Experiences

The paper introduces CompanionCast, a multi-agent framework that orchestrates specialized AI agents with multimodal detection, context caching, and spatial audio to enhance social presence and emotional sharing during shared media experiences, as validated by improved outcomes in pilot studies with soccer fans.

The Gist

Imagine you're sitting on your couch watching a thrilling soccer match. You're alone, but you wish you had a group of friends right there with you to scream "GOAL!" when the ball hits the net, to argue about a bad referee call, or to crack a joke when a player slips.

That's exactly what CompanionCast is trying to solve. It's a new system that uses a team of AI "friends" to watch videos with you, making the experience feel less lonely and more like a real social gathering.

Here is how it works, broken down into simple concepts:

1. The Cast of Characters (The Multi-Agent Team)

Instead of having just one robotic voice narrating the game, CompanionCast creates a team of three distinct AI personalities, each with their own job and voice:

• The Die-Hard Fan: This is your hype man. They are super emotional, cheering wildly for your team, and screaming when a goal is scored.
• The Analyst: This is the smart friend who knows the rules. They calmly explain why a play worked, pointing out tactical moves and statistics.
• The Comedian: This is the friend who likes to tease the other team. They make sarcastic jokes and keep the mood light, even when things get tense.

The Analogy: Think of it like a sports bar. You don't just have one guy talking; you have the loud fan in the corner, the guy with the stats in his head, and the joker making everyone laugh. CompanionCast brings all three of these people to your living room.

2. The "Eyes and Ears" (Event Detection)

The system doesn't just chat randomly. It has "eyes" watching the video and "ears" listening to the audio. It uses special technology to spot Key Moments.

• The Trigger: When the AI sees a goal, a foul, or a replay, it knows, "Okay, this is the moment! Time to speak up!"
• The Memory: It keeps a "rolling memory" of the last minute of the game. This ensures that if the Analyst mentions a player's name, the Comedian can immediately make a joke about that specific player, just like real friends do.

3. The "Sound Stage" (Spatial Audio)

This is a cool feature. When the AI speaks, it doesn't just come from your speakers as a flat sound. It uses spatial audio to place each character in a specific spot in your room.

• The Analogy: Imagine sitting in a circle with friends. The Die-Hard Fan is shouting from your left, the Analyst is whispering from your right, and the Comedian is talking from behind you. Your brain naturally separates them, making it feel like you are physically in a room with a group, not just listening to a podcast.

4. The "Editor" (The Evaluator Agent)

Before the AI friends speak to you, they have a secret "boss" or Editor Agent.

• How it works: The three friends generate a conversation, but the Editor checks it first. It asks: "Is this funny? Does the Analyst sound too robotic? Is the Comedian being too mean?"
• The Refinement: If the conversation isn't good enough, the Editor sends it back for a quick rewrite. This ensures the final chat feels natural, authentic, and high-quality before you hear it.

5. Did It Work? (The Pilot Study)

The researchers tested this with two soccer fans. They watched a clip alone, then watched the same clip with CompanionCast.

• The Result: The fans felt much less alone. They rated the experience as having a higher "social presence" (feeling like they were with others).
• The Feeling: They said the different voices and personalities made them feel like they were hanging out with a group of friends, rather than just watching a video with a narrator.

The Big Picture

CompanionCast is trying to bring the magic of "hanging out with friends" back to our solitary screen time. It realizes that watching a movie or a game isn't just about the content; it's about the reactions, the arguments, and the shared excitement. By using a team of specialized AI agents that talk to each other and to you, it turns a lonely viewing session into a lively, shared party.

In short: It's like having a virtual sports bar in your pocket, ready to cheer, analyze, and joke with you the second the ball hits the net.

Technical Summary

Here is a detailed technical summary of the paper "CompanionCast: Toward Social Collaboration with Multi-Agent Systems in Shared Experiences."

1. Problem Statement

Shared experiences, such as co-viewing sports or collaborative learning, are fundamental to social connection. However, modern media consumption is increasingly solitary. While AI companions exist, current systems face significant limitations:

• Single-Agent Design: Most systems rely on a single AI persona, which fails to replicate the complex dynamics of a group conversation.
• Lack of Social Awareness: Existing collaborative systems often lack the ability to manage multiple concurrent needs (e.g., emotional expression, sensemaking, pacing, and inclusion) required for authentic group interaction.
• Contextual Grounding: Systems often struggle to maintain "common ground" with users during fast-paced, dynamic events, leading to disjointed or irrelevant interactions.

The core challenge is creating an AI system that can orchestrate multiple specialized agents to simulate a realistic, socially aware group dynamic in real-time shared media consumption.

2. Methodology: The CompanionCast Framework

CompanionCast is a generalizable, LLM-powered multi-agent framework designed to orchestrate AI companions ($A = \{a_1, a_2, ..., a_n\}$) around multimodal video content. The system operates through four core functional components:

A. Multimodal Content Processing & Context Caching

• Input: The system ingests a multimodal stream ($S$) comprising video frames ($V$), audio ($A$), and metadata ($M$).
• Event Detection: It utilizes two annotation streams to identify triggers for interaction:
  1. Key Moments: High-level events (goals, fouls, corners) detected via the SoccerNet Dense Video Captioning dataset.
  2. Replay Segments: Temporal boundaries for replays detected via the SoccerNet Replay Grounding dataset.
• Rolling Context: A temporal context window ($C_t$) is maintained for all agents. For sports viewing, a 60-second look-back window ($\omega = 60s$) is cached to ensure agents have immediate grounding in recent events.

B. Multi-Agent Orchestration

The system instantiates specialized agents with distinct social roles and personality profiles ($p_i$), domain knowledge ($k_i$), and interaction styles ($s_i$). In the pilot study, three specific "fan" agents were created:

1. Die-Hard Fan: An enthusiastic supporter of the user's team, characterized by emotional expressiveness and celebratory language.
2. Analyst Fan: A tactical expert providing objective technical observations and performance commentary.
3. Comedian Fan: A sarcastic supporter of the opposing team, introducing playful antagonism and humor to create conversational tension.

Interaction Logic:

• Agents generate responses ($r_{i,t}$) conditioned on the user profile, shared context, and the history of all agent responses.
• Triggering: An orchestrator manages the transition from passive observation to active collaboration based on detected moments, scene changes, or user input.
• LLM Configuration: Fan agents use Claude Sonnet 4 (Temperature $\tau=0.7$) to encourage diversity, while the evaluator uses OpenAI GPT-4o (Temperature $\tau=0.2$) for consistent feedback.

C. Evaluator-Driven Refinement (LLM-as-a-Judge)

A meta-agent (Evaluator) assesses the generated dialogue across five dimensions: Relevance, Authenticity, Personality Consistency, Diversity, and Engagement.

• Process: The system executes a multi-round conversation protocol. Agents generate an initial response, receive quantitative scores (0–10) and qualitative feedback from the evaluator, and perform iterative refinements.
• Timing: During high-intensity moments (e.g., goals), $k=3$ refinement rounds are executed. For replays or user queries, $k=1$ or $2$ rounds are used to balance quality with latency.

D. Spatial Audio Rendering

To enhance co-presence (the feeling of being together), the system employs spatial audio:

• Each agent is assigned a distinct voice identity (via ElevenLabs TTS) and a specific stereo pan value ($l_i$).
• Audio is routed through a StereoPannerNode in a Web AudioContext to create left-right spatial separation.
• Original match audio is automatically muted during agent conversations to ensure intelligibility.

3. Key Contributions

1. Multi-Agent Social Framework: Introduction of CompanionCast, a novel architecture that moves beyond single-agent companions to orchestrate role-specialized agents for shared experiences.
2. Dynamic Grounding Mechanism: Implementation of a rolling context cache and event-triggered interaction model that allows agents to react proactively to real-time media events.
3. Quality Control Loop: A novel application of "AI-in-the-loop" quality control, where an evaluator agent refines multi-agent dialogue in real-time or near-real-time to ensure personality consistency and engagement.
4. Spatial Audio Integration: The integration of spatial audio positioning to differentiate agents and simulate physical co-presence, enhancing the social interpretability of the system.

4. Results (Pilot Study)

The system was evaluated via an in-person, within-subjects study with two adult soccer fans watching 5-minute clips under two conditions: Baseline (solitary viewing) and CompanionCast (viewing with AI companions).

• Perceived Social Co-presence: Both participants rated their sense of social co-presence at 4/5 and immersion at 3/5, indicating a moderate to high increase compared to solitary viewing.
• Emotional Sharing: Participants expressed high willingness to share emotions with the system (ratings of 4/5 and 5/5).
• Behavioral Engagement: Participants actively initiated interactions (P1: 2 messages; P2: 4 messages), demonstrating that the system successfully prompted user engagement.
• Qualitative Findings:
  • Role Differentiation: Users found the agents distinguishable and socially interpretable, appreciating the diversity of perspectives (tactical vs. emotional vs. humorous).
  • Social Lens: Users interpreted the agents as distinct social collaborators rather than a unified background voice, comparing them to human co-viewers.
  • Attention Maintenance: Agent reactions helped sustain attention and reduced the likelihood of missing key events.

5. Significance and Future Work

Significance:
CompanionCast demonstrates that multi-agent systems can effectively function as social collaborators in shared experiences when coordination, role differentiation, contextual grounding, and interaction modality are prioritized. It proves that distributing social functions across multiple personas is more effective than concentrating them in a single AI, and that multimodal cues (specifically spatial audio) are critical for interpretability.

Future Work:

• Broader Domains: Extending the framework beyond sports to movies, documentaries, and educational content.
• Larger Studies: Conducting studies with larger, more diverse participant groups to validate generalizability.
• Embodied Interaction: Exploring adaptive and embodied interaction paradigms (e.g., AR/VR integration as hinted in Figure 3 of the paper).
• Safeguards: Addressing challenges related to transparency, coordination complexity, and safety in human-agent collaboration.