Technical Summary: JoyAI-VL-Interaction

Problem Statement

Current large language and vision-language models (VLMs) operate primarily on a turn-based paradigm: they remain idle until prompted by a user, reacting only after a specific input is received. This structural limitation prevents them from handling real-world moments that require immediate, unprompted intervention, such as detecting a fire on a security feed, reacting to a fleeting expression in a video call, or assisting a shopper navigating changing app screens. While existing "real-time" omni-models (e.g., GPT-Realtime-2, Qwen3.5-Omni) reduce latency, they still optimize for conversational turn-taking rather than event-driven proactivity. Similarly, consumer video-call products (e.g., Doubao, Gemini) rely on external polling mechanisms or strict question-answer cycles, meaning they cannot perceive "now" or decide to speak until a user trigger or a fixed polling interval occurs. There is a lack of systems that continuously watch a visual stream, decide internally when to act, and maintain presence over unbounded time without waiting for a user prompt.

Methodology

The authors propose JoyAI-VL-Interaction, a paradigm shift from turn-based dialogue to real-time streaming interaction. The system is built on two core pillars: a proactive interaction model and a complete, deployable system architecture.

1. The Interaction Model (JoyAI-VL-Interaction)

• Architecture: Built upon the JoyAI-VL 1.0 base model (initialized from Qwen3-8B and Qwen3-VL ViT), the model is designed to be vision-first. Speech (ASR/TTS) is treated as a pluggable I/O module, decoupled from the core decision-making process.
• Action Space: At every second of a video stream, the model makes a discrete, learned decision to:
  1. Stay Silent: Continue watching (treated as a first-class action, not just the absence of speech).
  2. Respond: Generate a real-time reply to the user.
  3. Delegate: Offload complex, slow, or hard reasoning tasks to an asynchronous background model while maintaining presence.
• Video Encoding: To handle unbounded streams efficiently, the system uses AdaCodec. Instead of encoding every frame with full ViT tokens, it transmits only what prediction cannot explain. Predictable frames are encoded as compact P-tokens (motion + residuals), reducing token consumption by ~16x for predictable content and ensuring the context budget grows slowly over time.
• Training Strategy:
  • Data Construction: The authors created a dataset of 4M+ time-aligned streaming clips across six families: proactive alerting, time-aligned QA (backward/present/forward), counting, live commentary, casual chat, and delegation episodes.
  • Per-Second Supervision: The model is trained to predict the correct action (silence, response, delegate) for every second of the video. Silence is explicitly labeled as a positive target to prevent the model from speaking unnecessarily.
  • Loss Function: A weighted cross-entropy loss is used to handle the class imbalance where silence steps vastly outnumber response steps. Silence continuations are down-weighted, while response onsets are up-weighted.
  • Reinforcement Learning (RL): A GRPO-based RL stage optimizes the per-second policy using stream-level rewards, penalizing false alarms, mistimed responses, and degenerate "always-respond" behavior, while rewarding correct timing and effective delegation.

2. The System Architecture

The model is wrapped in a complete, open-source system designed for deployment:

• Dual-Loop Operation: The system runs two concurrent loops:
  • Real-time Loop: Handles ingestion, the interaction model's decision, and immediate TTS/UI output.
  • Asynchronous Loop: Handles delegated tasks via a Background Bridge. This bridge connects to any external API, agent, or model. It manages task IDs, timeouts, and result folding, allowing the model to stay present with the user while a complex task is processed in the background.
• Long-Horizon Memory: A hierarchical memory system (Short-term raw tokens, Mid-term text summaries, Long-term compressed blocks) allows the system to maintain context over hours of continuous video without exceeding context windows. This memory is optimized for prefix reuse in vLLM, enabling sub-second latency even with long histories.
• Modularity: The system is designed with "decision in the model, rest replaceable." Components like ASR, TTS, memory backends, and the background brain are pluggable, allowing users to swap in custom modules without rebuilding the stack.

Key Contributions

1. JoyAI-VL-Interaction Model: An 8B-scale, vision-first interaction model that learns to decide when to speak, stay silent, or delegate. It is the first open, vision-driven interaction model released with its training recipe and data.
2. Transferable Training Recipe: A data construction and training methodology that yields emergent capabilities (e.g., guiding users through changing app screens without specific training data, improvising lectures from slides) and transfers cleanly to new domains.
3. Complete Deployable System: A full-stack system supporting hours of continuous video with sub-second latency, featuring a background bridge for asynchronous delegation, hierarchical memory, and a visualization UI.
4. Open Release: The authors commit to releasing model weights, interaction data, and full system code (targeted for June 20, 2026), enabling the community to reproduce and build upon the work.

Results

The authors evaluated JoyAI-VL-Interaction against the in-app video-call assistants of Doubao and Gemini across six real-world streaming scenarios: monitoring/alerting, real-time counting, real-time translation, time awareness, live commentary, and long-horizon memory.

• Human Evaluation: In a head-to-head comparison with 58 cases rated by human experts on Quality and Timing:
  • JoyAI-VL-Interaction won 77.6% of comparisons against Doubao (vs. 5.2% loss).
  • It won 87.9% of comparisons against Gemini (vs. 1.7% loss).
• Scenario Performance:
  • Perfect Wins: The model achieved 100% win rates in Monitoring/Alerting against both baselines, and 100% in Real-time Counting and Translation against Gemini.
  • Timing Advantage: The model's primary advantage was in timing. Baselines often failed to react until a user prompt or polling interval passed, whereas JoyAI-VL-Interaction reacted at the exact moment of the event.
  • Emergent Capabilities: The model demonstrated abilities not explicitly trained, such as guiding a user through a shopping app interface and maintaining a strict 4-second commentary cadence, suggesting a general "watch-and-do" competence.
• Limitations: The authors acknowledge that on general turn-based chat quality and robustness to rare inputs, the 8B model is outperformed by the larger, mature models powering Doubao and Gemini. Occasional hallucinations in commentary were attributed to the model's compact scale and sparse commentary data.

Significance and Claims

The paper argues that interactivity should be scaled as a distinct capability of the model itself, rather than simulated by external triggers or turn detectors. The significance of this work lies in:

• Paradigm Shift: Moving VLMs from "submit a request and wait" to a "watch-and-do" mode of collaboration, where the AI is genuinely present in the physical world.
• Event-Driven Presence: Demonstrating that a compact model can outperform much larger, turn-based systems in time-critical, event-driven scenarios simply by being able to decide when to act.
• Open Foundation: By releasing the full stack (model, recipe, data, system), the authors aim to lower the barrier for the community to explore "always-present" assistants, AI glasses, and embodied intelligence, shifting the field from turn-based dialogue toward genuine streaming interaction.

The authors conclude that while the current release is an early stage with room for data scaling and robustness improvements, the emergence of unprompted, time-aware behavior validates the direction of scaling interactivity as a primary capability.