💻 computer science

Eddy-VL 1.9B: Structural Pruning and Layered Distillation for Edge-Deployable Multimodal Embedding

The paper introduces Eddy-VL 1.9B, a compressed multimodal embedding model designed for edge deployment that achieves 91.7% of its 2.13B teacher's performance while reducing parameters by 9.5% and latency by 10% through probe-driven structural pruning and layered knowledge distillation.

Original authors: HanYeong Cho, Changwoo Kim, Taeuk Chu, Jimin Park

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: HanYeong Cho, Changwoo Kim, Taeuk Chu, Jimin Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers can "see" and "read" at the same time, understanding a picture of a cat sitting on a mat just as well as they understand the words "cat on mat." This is the realm of multimodal AI, a branch of science where machines learn to connect images, text, and video into a single, shared language. To do this, these AI models act like giant translators, turning everything they see into a long list of numbers called an embedding. Think of an embedding as a unique fingerprint for a piece of content; if two fingerprints are very similar, the computer knows the content is related.

However, there's a catch. The smartest translators are usually the heaviest. They are like massive, cloud-based supercomputers that need a constant, high-speed internet connection to work. But what if you are in a secret basement, a police evidence room, or a remote field station where the internet is cut off? You need a translator that is smart enough to understand complex clues but small enough to fit on a local laptop or a handheld device. This is the challenge of edge deployment: making powerful AI work offline, quickly, and without needing a cloud connection.

Enter Eddy-VL 1.9B, a new model designed specifically for these "air-gapped" situations where privacy and speed are everything. The researchers behind this project started with a very smart, but very heavy, teacher model called Qwen3-VL-Embedding-2B. This teacher is like a brilliant professor with 28 layers of deep thinking, but it's too bulky to carry around in a backpack. The team asked a simple question: Can we shrink this professor down without making them forget how to teach?

They didn't just chop off random pieces; they used a clever strategy called structural pruning. Imagine the teacher's brain as a 28-story building. The researchers used a special "probe" to measure how much each floor was repeating the work of the floors right above and below it. They found four specific floors that were doing a lot of redundant work—like having four identical copy machines in a row when one would do. They removed these four floors, dropping the building from 28 stories to 24.

But here's the tricky part: when you remove floors from a building, the people living on the top floor might get lost. To fix this, the team used layered distillation. Think of this as a master architect (the original 28-story teacher) guiding a new, smaller architect (the 24-story student). The teacher doesn't just say "build it"; they show the student exactly how to think at every step. They used a technique called CKA (which measures how similar the thoughts of two layers are) to make sure the student's middle layers were thinking just like the teacher's, and they used a special "Matryoshka" method for the final answer, ensuring the student could give answers of different sizes depending on the need.

The result is Eddy-VL 1.9B. It's a model with about 1.93 billion parameters, weighing in at 3.85 GB—small enough to fit on many modern devices. In tests, this "shrunk" model managed to keep about 91.7% of the original teacher's intelligence. While the original teacher scored 68.9 on a massive test of 78 different tasks (called MMEB-V2), Eddy-VL scored 63.2. That might sound like a drop, but remember, the model lost four entire layers of thinking! Without the special teaching tricks, the score would have plummeted to 56.8. The distillation process successfully recovered 6.4 points of that lost ground.

The model also got a speed boost. Because it has fewer layers to process, it runs about 10% faster than the original, taking 136.4 milliseconds per image instead of 150.0 milliseconds. This might not sound like a huge difference, but in a real-world investigation where you are scanning thousands of seized photos offline, those extra seconds add up to hours of saved time.

The researchers were careful to note where the model still struggles. While it became nearly as good as the teacher at understanding complex relationships between words and images (scoring 86.1% on one test compared to the teacher's 86.4%), it still had a bit of trouble with a specific type of puzzle called Winoground, where it scored 6.8 compared to the teacher's 8.5. This suggests that while the model is excellent for general retrieval, some very tricky logical puzzles still require the full, unpruned brain.

Ultimately, Eddy-VL 1.9B isn't a magic bullet that solves every problem, but it is a powerful tool for a very specific job. It proves that you can take a massive, cloud-heavy AI and compress it into a lightweight, offline-ready package without losing too much of its soul. For investigators, forensic experts, and anyone working in places where the internet is a luxury they can't afford, this model offers a way to keep the lights on and the intelligence high, right on the edge of the network.

Technical Summary: Eddy-VL 1.9B

Problem Statement

The paper addresses the challenge of deploying high-performance multimodal embedding models in edge-deployable, air-gapped forensic and investigative settings. In these environments, cloud APIs are unavailable, latency is critical, and strict no-cloud policies exist. While the Qwen3-VL-Embedding-2B model serves as a leading open teacher for vision-language retrieval, its 28-layer text decoder and 4.26 GB checkpoint size are too heavy for these constrained edge scenarios. The goal is to compress the model to reduce parameter count and inference latency while preserving retrieval behavior and compositional understanding.

Methodology

The authors propose Eddy-VL 1.9B, a compressed multimodal embedding model derived from Qwen3-VL-Embedding-2B. The compression strategy involves two primary stages:

1. Probe-Driven Structural Pruning

Instead of training a new foundation model, the authors remove redundant layers from the existing teacher's text decoder.

Redundancy Detection: They extract hidden states from the last token of each of the 28 teacher decoder blocks using 1,000 calibration image-caption pairs.
Scoring: Adjacent-layer redundancy is scored using Linear Centered Kernel Alignment (CKA). The score $s(\ell)$ for a block is the sum of CKA with its predecessor and successor: $s(\ell) = \text{CKA}(X_{\ell-1}, X_\ell) + \text{CKA}(X_\ell, X_{\ell+1})$ .
Selection Strategy: Rather than simply removing the top 4 highest-scoring blocks (which would remove blocks 21–24), the authors analyze the balance of the CKA scores. They found that removing block 2 (high score but asymmetric neighbors) degraded early-depth transitions. Instead, they selected a set $D = \{3, 22, 24, 25\}$ to remove. This choice preserves the early-depth ramp and ensures a more symmetric removal of mid-to-late layers.
Result: The text decoder is reduced from 28 layers to 24 layers. The vision encoder (24 blocks) remains frozen.

2. Layered Knowledge Distillation

The pruned 24-layer student is trained to mimic the 28-layer teacher using a hole-covering teacher–student mapping.

Mapping: Removed blocks are "bridged" by mapping student layers to neighboring teacher layers. For example, student layer 3 maps to teacher layer 4; student layer 21 maps to teacher layer 23, etc.
Loss Functions:
- Mid-Layer Loss: For mapped intermediate layers, the authors use $1 - \text{CKA}$ on flattened attention maps to align the structural representation between teacher and student.
- Final-Layer Loss: The final embedding is aligned using MSE + Cosine loss with Matryoshka Representation Learning (MRL) dimensions $\{128, \dots, 2048\}$ .
Training Data: The distillation uses a mixture of two proprietary corpora:
- Korean COCO: ~410k image-caption pairs (image + Korean text).
- OurDataset: ~45k image-only samples (no text captions) covering forensic-relevant domains (documents, scenes, devices, etc.).

Key Contributions

Eddy-VL 1.9B: A released model with 1.93 billion parameters (3.85 GB in bf16), representing a ~9.5% reduction from the 2.13B teacher.
Novel Pruning Heuristic: A probe-driven method that prioritizes layer balance over raw redundancy scores to avoid collapsing early-depth representations, validated by ablation studies showing that naive top-k pruning fails on classification-heavy tasks.
Layered Distillation Framework: A specific distillation protocol using hole-covering mappings and attention-map CKA mid-losses, which outperforms hidden-only distillation and other mid-layer objectives (RCKA, STRUCTURE, mutual-kNN).
Edge Optimization: The model is explicitly designed for offline, air-gapped forensic retrieval, offering a ~10% reduction in forward latency.

Results

Evaluated on the MMEB-V2 benchmark (78 tasks) using the VLM2Vec framework:

Overall Performance:
- Teacher (Qwen3-VL-Embedding-2B): 68.9 (evaluated under the paper's specific protocol).
- Pruned Baseline (No Distillation): 56.8 (a loss of 12.1 points).
- Eddy-VL 1.9B: 63.2.
- Retention: Eddy-VL retains 91.7% of the teacher's performance under this protocol and recovers 6.4 points of the 12.1 points lost by pruning alone.
Compositional Probes:
- SugarCrepe: 86.1 (vs. Teacher 86.4).
- MR2-Bench: 24.5 (vs. Teacher 24.7).
- ARO: 59.5 (vs. Teacher 60.4).
- Winoground: 6.8 (vs. Teacher 8.5). The authors note this is the model's main weakness.
Efficiency:
- Forward latency reduced from 150.0 ms to 136.4 ms per image (~10% speedup) on an NVIDIA DGX Spark with FlashAttention-2.

Significance and Claims

The paper claims that Eddy-VL 1.9B demonstrates the effectiveness of structural pruning combined with layered distillation for creating edge-deployable multimodal retrievers.

Modest Claims: The authors explicitly state they do not claim parity with every published leaderboard configuration (e.g., the public Qwen leaderboard score of 73.2 vs. their re-evaluated 68.9). They focus on relative compression gains under a fixed protocol.
Target Use Case: The model is positioned as an embedder for candidate surfacing in forensic investigations, intended to be paired with a reranker for final ordering.
Limitations: The authors acknowledge a remaining ~8.3% relative gap to the teacher, the proprietary nature of the training data (preventing public redistribution), and the specific weakness in Winoground group scores.

The work provides a practical pathway for deploying large vision-language models in resource-constrained, offline environments without sacrificing the majority of their retrieval capabilities.