Modeling Cross-vision Synergy for Unified Large Vision Model

This paper introduces PolyV, a unified large vision model that achieves cross-vision synergy across images, videos, and 3D data through a sparse Mixture-of-Experts architecture with dynamic routing and a synergy-aware training paradigm, resulting in significant performance improvements over existing models.

The Gist

Imagine you have a team of experts in a room, but they all speak different languages and only know about one specific thing. One is a master of photos (static images), another is a master of movies (videos with motion), and a third is a master of 3D blueprints (spatial geometry).

In the past, if you asked them a question that required combining their skills—like "If I push this toy in the photo, where will it roll in the video?"—they would struggle. They would just talk past each other, or worse, the photo expert would forget everything they knew about photos because they were trying to learn about movies.

Enter PolyV (the "Poly" stands for many, and "V" for vision). It's a new kind of AI brain designed to make these experts work together seamlessly, creating a "synesthetic" experience where seeing a photo feels like understanding a video, and looking at a 3D model feels like walking through a room.

Here is how PolyV works, broken down into simple concepts:

1. The "Specialist Team" (The MoE Architecture)

Instead of one giant brain trying to do everything at once (which often leads to confusion), PolyV uses a Mixture of Experts (MoE).

• The Analogy: Think of PolyV as a high-end consulting firm. When a client (you) walks in with a question, a Smart Manager (the Router) immediately looks at the problem and calls the right specialist.
  • If you ask about a still photo, the Photo Expert takes the lead.
  • If you ask about a car crash in a video, the Video Expert jumps in.
  • If you ask about the layout of a room, the 3D Expert steps forward.
• The Magic: Unlike old models where experts worked in silos, PolyV's manager allows them to chat. The Photo Expert can say, "Hey, based on how light hits this object, I bet the 3D Expert knows the shape is actually a cube." This cross-talk is the "synergy."

2. The "Translator" (Training Strategy)

To get these experts to understand each other, the researchers didn't just throw them in a room and hope for the best. They used a two-step training camp:

• Step 1: Specialized Boot Camp (Pre-training)
  First, each expert gets rigorous training only on their own subject. The Photo Expert learns to recognize textures; the Video Expert learns to predict motion; the 3D Expert learns geometry. They become masters of their own domain.

• Step 2: The "Synergy" Workshop (Fine-tuning)
  This is where the magic happens. The researchers introduce a special training method using Knowledge Distillation.

  • The Analogy: Imagine the Photo Expert is trying to guess what happens next in a video. They don't just guess; they peek at the "answer key" provided by a super-smart Video Teacher. The Photo Expert learns to say, "Ah, because the ball is moving left in the photo, it will likely roll off the table in the video."
  • They also use Scene Graphs (like a family tree for objects). They teach the AI to ask: "Is the baby pushing the toy in the photo the same as in the video?" This forces the AI to link the static image to the dynamic video and the spatial 3D model.

3. The Result: "Synesthetic Vision"

The paper calls this Cross-Vision Synergy. In human terms, it's like synesthesia (a condition where senses blend, like "hearing" colors).

• Old AI: Sees a photo of a golf ball and says, "That's a ball."
• PolyV: Sees the photo and says, "That's a ball, and because of the grass texture and the club's angle, I know exactly where it will land, how fast it will roll, and how it will look from a 3D perspective."

Why Does This Matter?

Current AI models are often "one-trick ponies." If you train them on 3D data, they forget how to understand videos. If you train them on videos, they get bad at spatial reasoning.

PolyV solves this by:

1. Not forgetting: It keeps its specialized knowledge intact while learning new skills.
2. Connecting the dots: It uses what it knows about 3D space to understand a 2D photo better, and uses motion from videos to understand static images.
3. Being efficient: It only "wakes up" the experts it needs, saving computer power.

In a Nutshell

PolyV is the first AI that doesn't just see images, videos, and 3D worlds separately. It feels them all together. It's like upgrading from a person who can only read a map, to a person who can read the map, feel the terrain, and predict the weather all at the same time. This makes it incredibly powerful for things like self-driving cars, robotics, and virtual reality, where understanding the world in 3D, over time, and from different angles is crucial.

Technical Summary

1. Problem Statement

Current Large Vision Models (LVMs) have shifted toward unified architectures capable of processing images, videos, and 3D data. However, existing models primarily achieve functional integration (handling multiple modalities in one system) rather than cross-vision synergy.

• The Gap: Existing models fail to perform "synesthetic reasoning," where priors from one modality (e.g., temporal motion from video or 3D geometry) actively inform and enhance reasoning in another (e.g., static image spatial understanding).
• Limitations of Current Approaches:
  • Architectural: Most models use shared encoders or simple feature concatenation, which isolates modalities and prevents dynamic, bidirectional exchange of priors.
  • Training: Current strategies rely on independent fine-tuning or incremental training, leading to catastrophic forgetting of previously learned modalities and a lack of deep alignment between object-level attributes and relational dynamics across modalities.

2. Methodology: PolyV

The authors propose PolyV, a unified LVM designed to achieve cross-vision synergy at both the architectural and training levels.

A. Architecture: Sparse Mixture-of-Experts (MoE)

PolyV utilizes a sparse MoE architecture coordinated by a dynamic router to balance specialization and interaction.

• Universal Vision Encoder: A single encoder processes images, video (as frame sequences with temporal embeddings), and 3D data (with injected 3D coordinates/depth) into a unified token stream.
• Dynamic Router: A learnable router assigns tokens to specific experts.
• Expert Specialization: The model employs multiple Feed-Forward Network (FFN) experts. Unlike standard MoE, these experts are designed to specialize in modality-specific priors (e.g., temporal dynamics for video, geometric structure for 3D) while the router enables bidirectional interaction. This allows the model to leverage complementary priors during inference (e.g., using 3D geometric knowledge to solve a video spatial reasoning task).

B. Training Strategy: Synergy-Aware Paradigm

PolyV employs a two-stage training process to foster deep cross-modal alignment:

1. Stage 1: Modality-Specific Pretraining

   • Alignment Pre-training: Establishes the initial connection between the vision encoder and the LLM backbone using 3D data.
   • Instruction-Following: Experts are trained independently on modality-specific instruction data (images, videos, 3D) to build strong foundational representations for each domain without interference.

2. Stage 2: Synergy-Aware Fine-tuning

   • Coarse-Grained Tuning (Knowledge Distillation):
     • Introduces <synergy> tokens as cognitive mediators.
     • Uses Knowledge Distillation from strong single-modality foundation models (Video Foundation Model like V-JEPA 2 for temporal priors; 3D Foundation Model like VGGT for spatial priors).
     • The model learns to align its latent synergy tokens with the temporal and spatial features extracted by these teacher models, forcing the LLM to internalize cross-modal priors.
   • Fine-Grained Tuning (CSQA Dataset):
     • Constructs a Cross-Vision Synergy Question-Answer (CSQA) dataset using scene graphs.
     • Generates questions requiring reasoning about object-level (spatial consistency, motion continuity) and relation-level (interaction dynamics, viewpoint changes) correspondences across image-video and image-3D pairs.
     • Trains the model to answer these questions, enforcing precise alignment of object attributes and relationships across modalities.

3. Key Contributions

1. Conceptual Shift: Proposes Cross-Vision Synergy as the next frontier for LVMs, moving beyond functional unification to synesthetic reasoning where modalities mutually refine each other.
2. Architectural Innovation: Designs a Dynamic Router-guided MoE architecture that explicitly disentangles modality-shared and modality-specific features, enabling adaptive expert allocation and bidirectional prior exchange.
3. Training Framework: Introduces a novel Synergy-Aware Training pipeline combining:
   • Distillation from specialized foundation models (Video/3D) to inject priors.
   • Fine-grained alignment via scene-graph-based CSQA data to model object/relation dynamics.
4. Comprehensive Evaluation: Demonstrates that PolyV achieves state-of-the-art performance across 10 benchmarks spanning image, video, and 3D understanding, including tasks requiring complex spatial and temporal reasoning.

4. Experimental Results

PolyV was evaluated on 10 diverse benchmarks, including MMStar, 3DSRBench, VSI-Bench, VideoMME, and Open-EQA.

• Performance Gains: PolyV consistently outperforms its backbone (Qwen2.5-VL-7B) and existing state-of-the-art models (e.g., LLaVA-OV, InternVL3, SpaceR, Ross3D).
  • Achieved an average improvement of over 10% over the backbone.
  • Notable gains in spatial reasoning (e.g., +15.0% on 3DSRBench) and temporal reasoning (e.g., +19.7% on VSI-Bench).
• Ablation Studies:
  • MoE Necessity: Removing the MoE structure (dense model) resulted in significant performance drops, confirming the value of expert specialization.
  • Training Strategy: Combining both coarse-grained (distillation) and fine-grained (CSQA) tuning yielded the best results; using either alone was suboptimal.
  • Foundation Models: Integrating both Video and 3D foundation models provided complementary guidance, with VideoFM boosting temporal tasks and 3DFM boosting spatial tasks.
• Routing Analysis: Visualization of expert routing showed that PolyV adaptively allocates experts based on task priors (e.g., specific experts dominate for spatial reasoning, others for temporal dynamics), confirming effective cross-vision synergy.
• Catastrophic Forgetting: Unlike models that suffer from forgetting when learning new modalities, PolyV maintained high performance on original image/video tasks while acquiring 3D capabilities.

5. Significance

• Human-Like Reasoning: PolyV moves LVMs closer to human-like "synesthetic" perception, where understanding a static image can be enhanced by "imagining" motion or 3D structure, and vice versa.
• Unified Framework: It provides a scalable, extensible framework for structured multimodal reasoning that does not require separate models for different modalities.
• Future Impact: The ability to reason across modalities is critical for downstream applications such as robotics, autonomous driving, and embodied AI, where agents must seamlessly integrate visual, spatial, and temporal cues to navigate and interact with the real world.

In conclusion, PolyV establishes a new paradigm for Unified Large Vision Models by explicitly modeling the synergistic relationships between visual modalities, achieving superior reasoning capabilities through a specialized MoE architecture and a synergy-aware training regimen.