Technical Summary: From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

Problem Statement

While Multimodal Large Language Models (MLLMs) have achieved remarkable progress in unifying vision and language, existing surveys often treat these modalities in isolation or focus on specific downstream tasks (e.g., object detection, segmentation) rather than perception as a unified capability. Current literature lacks a systematic examination of vision-language perception from an integrated perspective—one that treats the extraction, localization, and reasoning about structured semantic information from raw visual signals as an intrinsic, inseparable modality. This fragmentation obscures the paradigm-level transitions in how MLLMs perceive the world, hindering a holistic understanding of their potential to achieve Artificial General Intelligence (AGI).

Methodology and Taxonomy

The paper proposes the first systematic survey of unified vision-language perception in MLLMs, organized around a five-stage taxonomy that traces the evolution from structure-driven optimizations to synergy-driven integration. The authors categorize representative works not strictly by chronology, but by their primary contribution to the evolution of perception capabilities:

Stage I: Encoder-Centric Optimization

Early approaches focused on modifying the vision encoder to enhance local, fine-grained perception.

• Region-Aware Modules: Methods introduce universal Region-of-Interest (ROI) proposals, query-aware encoders, or projector/connector-level optimizations to shift from global image representations to region-level understanding (e.g., MG-LLaVA, ChatRex, Groma).
• Integrated Perception Subnetworks: Approaches treat the encoder as a subnetwork, employing specialized task encoders (e.g., for OCR, detection) or multiple visual-modal encoders (e.g., depth, segmentation) to fuse diverse visual perspectives.

Stage II: Decoder-Centric Optimization

This stage marks a shift from region-level to pixel-level perception by introducing auxiliary decoders.

• Auxiliary Decoders: Works like LISA pioneered end-to-end segmentation within MLLMs by generating special tokens decoded into pixel masks. Subsequent research expanded this to temporal contexts (video) and multi-granularity perception.
• Multi-Decoder Architectures: Frameworks equip MLLMs with multiple parallel task-specific heads (e.g., box, mask, point decoders) to handle diverse perception tasks within a unified backbone.
• Specialized Decoding Strategies: Methods leverage structured text outputs (e.g., coordinates, masks) either as direct task results or as prompts to guide downstream modules like SAM.

Stage III: Dynamic Perception via Adaptive Processing

Moving beyond static, single-pass encoding, this stage introduces adaptive, context-aware strategies where the model interacts with images multiple times.

• External Tool Scheduling: MLLMs act as coordinators, invoking external tools (OCR, detectors) via fixed pipelines or dynamic expert selection (e.g., LLaVA-Plus, TACO).
• LLM-Centric Routing: The model autonomously routes perception processes, performing step-by-step visual reasoning, multi-round iterative search (e.g., ZoomEye), or detail enhancement via re-encoding.
• Code-as-Policy: MLLMs generate executable Python code to perform perception tasks, leveraging the flexibility of programming for complex visual reasoning (e.g., ViperGPT, PyVision).

Stage IVA: Architecture-Free Strategies

This stage focuses on enhancing perception without modifying model architecture, primarily through advanced training strategies.

• Instruction-Based Strategies: Utilizing instruction tuning and specific data formats (e.g., location tokens) to teach spatial reasoning and grounding (e.g., Kosmos-2, Shikra).
• RL-Based Strategies: Following the success of models like DeepSeek-R1, methods employ Reinforcement Learning (specifically Group Relative Policy Optimization, GRPO) with verifiable, rule-based rewards to improve performance on detection, segmentation, and reasoning tasks without architectural changes.

Stage IVB: Towards Unified Perception

The final stage envisions a convergence of instruction, adaptivity, and reinforcement learning into a unified framework.

• Convergence Signs: Emerging models (e.g., OpenAI o3) demonstrate "thinking with images," combining planning, dynamic tool use, and code execution in a closed loop.
• Emerging Agents: New research directions focus on agents that can autonomously decide when to zoom, crop, or re-evaluate visual inputs based on task needs, moving toward a unified perception-reasoning cycle.

Key Contributions

1. First Systematic Survey: The paper provides the inaugural systematic analysis of vision-language perception in MLLMs as an intrinsic, unified capability, distinguishing itself from prior surveys that focus on isolated tasks or broad architectural overviews.
2. Five-Stage Taxonomy: It establishes a comprehensive framework tracing the paradigm evolution from encoder/decoder-centric designs to dynamic, adaptive, and architecture-free strategies, culminating in unified perception.
3. Identification of Challenges and Directions: The authors identify critical bottlenecks, including the limitations of conventional evaluation metrics, the "generalization vs. fine-tuning" dilemma, and the need for general, fine-grained reward functions and cost-aware scheduling.

Results and Claims

The paper does not present new experimental results but synthesizes the state-of-the-art to demonstrate a clear trajectory:

• Granularity Evolution: Perception capabilities have evolved from image-level understanding to region-level, and finally to pixel-level and dynamic, iterative processing.
• Paradigm Shift: The field is transitioning from static, structure-driven modifications to dynamic, synergy-driven frameworks where perception emerges from integrated strategies rather than isolated modules.
• Emergence of "Thinking" Models: Recent developments (e.g., o3, Active-o3) suggest a move toward agents that can actively manipulate visual inputs (zoom, crop) and reason over them in a closed loop, mimicking human-like perception.

Significance and Future Outlook

The paper argues that to achieve truly general, unified multimodal intelligence, the field must move beyond isolated task optimization. The significance of this survey lies in providing a foundational understanding and an actionable roadmap for researchers.

The authors highlight three critical challenges for the next generation of perception-centric agents:

1. Evaluation: Traditional proxy tasks (static detection/segmentation) are becoming saturated and may not reflect real-world utility; new evaluation paradigms are needed.
2. Generalization vs. Fine-tuning: As foundational models improve, task-specific fine-tuning yields diminishing returns, suggesting a need for methods that leverage inherent generalization capabilities.
3. Unified Representation and Rewards: There is a pressing need for task-agnostic, hierarchically fine-grained reward frameworks and a unified representation space that bridges text, regions, and pixels without relying on prohibitively expensive data curation.

Ultimately, the survey posits that the path to AGI in multimodal systems lies in synergy—integrating instruction, adaptivity, and reinforcement learning to create agents capable of open-ended, perception-centric reasoning.