Efficient Universal Perception Encoder

1. Problem Statement

Deploying versatile AI on smart edge devices is challenging due to limited computational resources and the need to handle multiple tasks simultaneously (e.g., image understanding, dense prediction, and vision-language modeling).

• The Trade-off: Current foundation vision encoders are typically "domain experts." For instance, models like CLIP or SigLIP excel at image-text alignment but struggle with dense prediction (segmentation/depth), while models like DINO or SAM excel at spatial tasks but lack vision-language capabilities.
• The Limitation of Existing Solutions:
  • Single Encoders: Choosing one specific encoder forces a performance trade-off on other tasks.
  • Agglomerative Methods: Previous attempts to combine multiple teachers into one student (e.g., RADIO, DUNE) via direct distillation work well for large models but fail when scaled down to efficient backbones (e.g., ViT-B). These methods suffer because efficient encoders lack the capacity to directly absorb and unify diverse feature representations from multiple large teachers.

2. Methodology: The EUPE Pipeline

The authors propose a three-stage distillation pipeline based on the principle of "Scaling Up, then Scaling Down." Instead of directly distilling multiple teachers into a small student, they introduce a heavy "Proxy Teacher" to unify knowledge first.

Stage 1: Multi-Teacher Distillation (Scaling Up)

• Goal: Create a large Proxy Model (e.g., 1.9B parameters) that learns a universal representation.
• Process: The proxy model is trained via knowledge distillation from multiple domain-expert foundation models simultaneously.
• Teachers Used:
  • PEcore: For zero-shot image classification and retrieval.
  • DINOv3: For dense prediction (spatial/structural understanding).
  • PElang: For vision-language modeling (crucial for OCR and VLM tasks).
• Input: Label-free images passed to all teachers and the proxy at native resolutions.
• Loss: Cosine similarity for class tokens; a combination of cosine similarity and Smooth L1 loss for patch tokens. Feature normalization (mean/std subtraction) is applied to stabilize training across diverse teachers.

Stage 2: Fixed-Resolution Distillation (Scaling Down - Part 1)

• Goal: Transfer the unified knowledge from the Proxy Model to the target Efficient Student (e.g., ViT-B, ViT-S, ConvNext).
• Process: The student is distilled from the Stage 1 Proxy model.
• Configuration: Images are resized to a fixed resolution (256×256). This allows for a longer training schedule, which is computationally efficient and helps the student learn the universal representation effectively.

Stage 3: Multi-Resolution Finetuning (Scaling Down - Part 2)

• Goal: Adapt the student to handle various downstream task resolutions.
• Process: The student (initialized from Stage 2) is finetuned using an image pyramid (e.g., 256, 384, 512).
• Mechanism: The teacher and student randomly select different scales from the pyramid independently for each iteration. This forces the student to learn representations at different granularities, ensuring robustness for tasks requiring different input resolutions.

3. Key Contributions

1. Scaling-Up-Then-Scaling-Down Recipe: The paper identifies that direct multi-teacher distillation fails for efficient models. The introduction of a heavy Proxy Teacher acts as a necessary intermediate step to unify diverse knowledge before compressing it into an efficient student.
2. Efficient Universal Perception Encoder (EUPE) Family: A zoo of models (ViT and ConvNext variants ranging from Tiny to Base) that achieve on-par or superior performance compared to domain-specific experts of the same size.
3. Comprehensive Ablation Studies:
   • Teacher Selection: Demonstrated that combining PEcore, DINOv3, and PElang yields the best balance, whereas adding SigLIP2 (another CLIP-style model) caused degradation due to feature incompatibility.
   • Pipeline Necessity: Proved that skipping Stage 1 (direct distillation) or Stage 3 (multi-resolution) significantly hurts performance in VLM or dense prediction tasks, respectively.
   • Data Quality: Showed that the LVD-1689M dataset yields better results than larger but lower-quality datasets like MetaCLIP.

4. Experimental Results

The EUPE models were benchmarked against state-of-the-art (SOTA) domain experts and previous agglomerative methods (RADIOv2.5, DUNE) across three domains:

• Image Understanding (Zero-shot/KNN): EUPE-ViT-B achieves 79.7% (IN1k-ZS) and 84.1% (IN1k-KNN), matching or exceeding experts like PEcore and SigLIP2.
• Vision-Language Modeling (VLM):
  • Outperforms dense prediction experts (DINOv3) on all VLM tasks.
  • Surpasses VLM experts (SigLIP2, PEcore) on Realworld (55.5 vs 52.9), POPE (85.9 vs 85.8), and GQA (67.3 vs 65.9).
  • Maintains competitive performance on OCR (TextVQA) and Knowledge (SQA) tasks.
• Dense Prediction:
  • SPair: 51.3 (beating DINOv3's 51.3 and RADIO's 48.7).
  • ADE20k: 52.4 mIoU (outperforming DINOv3's 51.8).
  • NYUv2: 0.391 RMSE (competitive with DINOv3).
• Comparison to Agglomerative Methods: EUPE significantly outperforms RADIOv2.5 and DUNE across almost all benchmarks, particularly in VLM and dense prediction, proving the efficacy of the proxy teacher approach.
• Feature Visualization: PCA projections show EUPE features possess both semantic coherence (like DINO) and fine-grained sensitivity (unlike DINO), while avoiding the semantic noise seen in pure text-image models.

5. Significance

• Edge AI Enablement: EUPE provides a practical solution for running versatile, multi-task AI on resource-constrained edge devices (smartphones, AR/VR) without needing to load multiple large models.
• Paradigm Shift: It challenges the notion that efficient models must be specialized. By using a proxy teacher, it demonstrates that "universal" capabilities can be successfully distilled into compact architectures.
• Open Science: The authors commit to releasing the full family of EUPE models and code, fostering further research into universal perception and efficient distillation.

In summary, EUPE solves the "efficiency vs. versatility" dilemma in computer vision by introducing a Proxy Teacher architecture, enabling small models to match the performance of large, specialized experts across image understanding, dense prediction, and vision-language tasks.