🤖 machine learning

UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning

UNIEGO introduces a hierarchical multi-teacher distillation framework that employs representation-specific Proxy models and a Selective Proxy Distillation strategy to unify diverse knowledge from multiple viewpoints, modalities, and foundation models into a single, state-of-the-art egocentric video encoder.

Original authors: Wenhao Chi, Arkaprava Sinha, Dominick Reilly, Hieu Le, Srijan Das

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenhao Chi, Arkaprava Sinha, Dominick Reilly, Hieu Le, Srijan Das

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to cook a complex dish, but you are wearing a blindfold and only have a tiny peephole in front of your face. You can see the pot, but you can't see your hands, the ingredients on the counter, or the person standing next to you. This is exactly the problem computers face when trying to understand videos taken from a person's point of view (called egocentric video). The camera sees a narrow, shaky, and often blocked view, missing crucial details about the person's body or the surrounding room.

The paper introduces a new system called UNIEGO to solve this. Think of UNIEGO as a master chef who wants to learn the perfect recipe, but instead of just looking through that tiny peephole, they have access to a team of nine expert teachers.

Here is how UNIEGO works, broken down into simple steps:

1. The Problem: Too Many Teachers, Too Many Languages

Usually, if you want to learn from many teachers, you just ask them all to teach you at once. But in this case, the teachers are speaking different "languages" and looking at the world from different angles:

The Viewpoint Gap: Some teachers are wearing the same camera as you (Egocentric), while others are standing across the room watching you (Exocentric).
The Modality Gap: Some teachers see in standard color (RGB), some see in 3D depth (like a night-vision sensor), and some see only the skeleton of your bones.
The Model Gap: Some teachers are giant, pre-trained AI models (Foundation Models) that know a lot about the world, while others are specialized models for specific tasks.

If you try to listen to all nine teachers at the same time, it's like being in a room where everyone is shouting in different languages. The student gets confused, the instructions clash, and they learn nothing. This is called "conflicting gradients."

2. The Solution: The "Proxy" Translators

UNIEGO doesn't let the student talk to the teachers directly. Instead, it hires a team of Proxies.

Think of these Proxies as interpreters.

Each teacher (the skeleton expert, the depth camera, the giant AI) teaches one specific Proxy.
Crucially, every Proxy speaks the same language as the final student: they all look at the video through the same "egocentric" peephole.
The teacher teaches the Proxy their specific knowledge (e.g., "Here is how the skeleton moves"), and the Proxy translates that into a format the student can understand.

By the end of this first stage, the student has a pool of nine interpreters, all speaking the same language, but each holding a piece of the puzzle from a different expert.

3. The Smart Filter: "Selective Proxy Distillation"

Now comes the tricky part. Even with interpreters, sometimes an interpreter might be wrong or confused about a specific moment in the video. If the student listens to a confused interpreter, they will learn the wrong thing.

UNIEGO uses a smart filter called Selective Proxy Distillation (SPD).

For every single moment in the video, UNIEGO asks the interpreters: "Who is confident and correct right now?"
It ignores the interpreters who are unsure or wrong.
It only listens to the top few interpreters who are sure of their answer.

It's like a student in a classroom who only raises their hand to listen to the teacher who is 100% sure of the answer, ignoring the ones who are guessing. This prevents the student from getting confused by bad advice.

4. The Smooth Start: "Proxy Merging"

Before the student even starts learning from the interpreters, UNIEGO gives them a "head start." It takes the knowledge from all the interpreters and blends them together mathematically to create a starting point that is already pretty good. This places the student in a "safe zone" where learning is easier and less likely to go off the rails.

The Result

The final result is UNIEGO, a single AI model that:

Only needs one camera (the wearable one) to work during real-life use.
Knows more than any single camera could ever see, because it was trained on the combined wisdom of depth, skeletons, and outside views.
Performs better at recognizing actions, finding videos, and cutting up video clips than any previous method that tried to combine these views directly.

In short, UNIEGO is a master student that learns by hiring a team of translators to turn a chaotic room of shouting experts into a clear, single lesson, ensuring it only listens to the best advice at the right time.

Technical Summary: UNIEGO

Problem Statement

Egocentric video understanding is fundamentally constrained by the narrow field of view and persistent self-occlusions inherent to wearable cameras. A single model trained solely on egocentric RGB data often fails to capture the full richness of human action, lacking access to complementary geometric structures (depth, skeleton) and broader scene context available through exocentric viewpoints. Furthermore, the rich representational knowledge encapsulated within large-scale foundation models (FMs) and diverse modalities remains largely untapped for egocentric tasks.

While multi-teacher knowledge distillation offers a potential solution, existing approaches struggle in the egocentric setting due to two primary challenges:

Heterogeneity: Teachers vary significantly in architecture (e.g., graph-based skeleton models vs. video encoders), input modalities (RGB, depth, skeleton), and viewpoint (egocentric vs. exocentric).
Conflicting Gradients: Naively distilling from these heterogeneous teachers into a single student creates incompatible feature spaces, leading to conflicting gradient signals that degrade optimization and prevent the student from effectively bridging modality and viewpoint gaps.

Methodology: Hierarchical Multi-Teacher Distillation

The authors propose UNIEGO, a unified egocentric encoder trained via a two-level hierarchical distillation framework. This approach avoids direct distillation from heterogeneous teachers, instead interposing a layer of representation-specific Proxy models to mediate knowledge transfer.

Level-I: Proxy Learning

The first stage translates diverse teacher knowledge into a homogeneous egocentric space.

Architecture: A pool of $R$ proxy models is trained, where each proxy shares the same egocentric architecture (e.g., TimeSformer) as the final student.
Process: Each proxy $P_r$ independently distills knowledge from a specific teacher $T_r$ (spanning RGB, depth, skeleton, ego, and exo viewpoints, including FMs like DINOv2 and SigLIP).
Objective: The proxies learn to map heterogeneous teacher features into a consistent egocentric embedding space, effectively bridging the modality and viewpoint gaps. This stage uses feature-level distillation (cosine distance) combined with action classification loss.

Level-II: Selective Proxy Distillation (SPD)

The second stage consolidates the knowledge from the proxy pool into the final UNIEGO model.

Proxy Merging Initialization: To stabilize optimization, UNIEGO is initialized not randomly, but as a learned convex combination of the proxy parameters ( $\theta^*_{merge} = \sum \alpha^*_r \theta_r$ ). This places the unified model in a well-conditioned, flat region of the loss landscape before distillation begins.
Selective Proxy Distillation (SPD): Rather than distilling from all proxies simultaneously, SPD adaptively selects a subset of reliable proxies for each training sample.
- Selection Criteria: A proxy is considered a candidate only if it correctly predicts the action class. Among correct candidates, the top- $k$ proxies with the lowest cross-entropy loss (highest confidence) are selected.
- Suppression: If no proxy is correct for a sample, distillation is skipped for that sample to prevent the absorption of erroneous supervision.
- Loss: The selected proxies provide both feature-level and logit-level distillation signals to the student.

Key Contributions

UNIEGO Framework: Introduction of a unified egocentric encoder trained with nine teachers across multiple modalities, viewpoints, and foundation models, utilizing representation-specific proxies as structured mediators.
Three Tightly Coupled Components:
- Proxy Learning: Converts heterogeneous supervision into a pool of architecturally homogeneous egocentric proxies.
- Selective Proxy Distillation (SPD): A sample-wise mechanism that dynamically routes supervision from the most reliable proxies, mitigating conflicting gradients.
- Proxy Merging: A learned convex initialization that stabilizes the optimization landscape.
State-of-the-Art Performance: Demonstration that structured, proxy-mediated knowledge transfer yields richer representations than direct multi-teacher supervision.

Experimental Results

UNIEGO was evaluated on three challenging ego-exo benchmarks: EgoExo-Fitness, Assembly101, and EgoExo4D.

Action Recognition: UNIEGO achieved state-of-the-art performance under egocentric inference (using only egocentric video at test time).
- It outperformed the TimeSformer backbone by +4.4%, +3.1%, and +1.2% on the three respective datasets.
- It surpassed the strongest distillation baseline ( $\pi$ -ViT) by +4.6% and +2.9% on EgoExo-Fitness and Assembly101.
- Crucially, it outperformed naive multi-teacher distillation baselines, confirming the necessity of the proxy-mediated approach.
Video Retrieval & Segmentation: UNIEGO demonstrated superior performance in video retrieval (measured by mAP and Recall@1) and temporal action segmentation (F1 scores, Edit distance, and Accuracy) compared to baselines. Notably, naive multi-teacher distillation degraded segmentation performance relative to scratch baselines, whereas UNIEGO improved it.
Generalization: The framework proved robust across different backbone architectures, including compact models like UniFormer-S (22M parameters), showing effectiveness for resource-constrained deployment.

Significance and Claims

The paper claims that UNIEGO successfully addresses the "ill-posed" nature of learning a unified egocentric representation from heterogeneous sources. By introducing proxies as mediators, the framework:

Dissolves the Representational Gap: Projections of heterogeneous teachers into a homogeneous proxy space significantly increase the similarity between teacher representations (verified via Centered Kernel Alignment).
Mitigates Conflicting Gradients: The combination of proxy learning and selective distillation transforms a conflicting optimization landscape into a coherent one. Analysis shows that SPD significantly reduces gradient conflict rates compared to direct multi-teacher distillation.

The authors conclude that UNIEGO enables an egocentric encoder to "see further, know more, and generalize better" without requiring additional camera streams at inference time. They acknowledge a limitation in their current proxy selection strategy (relying on a small-loss heuristic) and suggest that a learned, adaptive selection mechanism is a promising direction for future work.