💬 NLP

READ: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling

This paper introduces READ, a parameter-efficient transfer learning framework that combines a novel recurrent adapter for temporal modeling with a partial video-language alignment objective to significantly outperform existing fine-tuning strategies on low-resource video-language tasks.

Original authors: Thong Nguyen, Xiaobao Wu, Xinshuai Dong, Khoi Le, Zhiyuan Hu, Cong-Duy Nguyen, See-Kiong Ng, Luu Anh Tuan

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thong Nguyen, Xiaobao Wu, Xinshuai Dong, Khoi Le, Zhiyuan Hu, Cong-Duy Nguyen, See-Kiong Ng, Luu Anh Tuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart library of books (a pre-trained AI model) that knows almost everything about the world. However, this library is so massive that it takes up an entire warehouse, and every time you want to teach it a new, specific skill—like summarizing a cooking video or finding the exact moment a proposal happens in a clip—you usually have to rewrite the entire library. This is expensive, slow, and requires a lot of storage space.

The paper introduces a clever, lightweight solution called READ (Recurrent Adapter) combined with a new way of checking work called PVLA. Here is how it works, broken down into simple concepts:

1. The Problem: The "Heavy" Library

Current methods try to teach the AI new skills by retraining the whole massive library.

The Issue: If you have a small amount of data (a "low-resource" scenario), trying to rewrite the whole library often makes the AI confused or unstable. Plus, you end up needing a separate, massive warehouse for every single new skill you teach it.

2. The Solution: The "Sticky Note" System (Adapters)

Instead of rewriting the whole library, the authors suggest adding tiny "sticky notes" (called Adapters) to the existing books.

How it works: You freeze the original library (so it stays perfect) and only train these tiny sticky notes. This saves huge amounts of space and money.
The Flaw in Old Sticky Notes: Previous "sticky notes" were too simple. They looked at video frames and words as isolated items, like looking at a single photo of a girl and a single word "proposal" without understanding the story connecting them. They missed the timeline.

3. The Innovation: The "Time-Traveling" Sticky Note (READ)

The authors created a new type of sticky note called READ (Recurrent Adapter).

The Analogy: Imagine a regular sticky note is a snapshot. A READ sticky note is like a flipbook animation.
What it does: It doesn't just look at one frame or one word; it looks at the sequence. It understands that the girl's expression after the proposal is different from her expression before it. It captures the flow of time, allowing the AI to understand the story's timeline without needing to retrain the whole library.

4. The Quality Check: The "Partial Match" Game (PVLA)

When teaching the AI with limited data, there's a risk of the "sticky notes" getting confused by noise or irrelevant information.

The Problem: A video might show a whole scene (a kitchen, a bike, a person), but the text might only talk about one specific part (tightening a bolt). Old methods tried to force a perfect 1-to-1 match between every word and every video frame, which is impossible and confusing.
The Solution (PVLA): The authors introduced Partial Video-Language Alignment (PVLA).
The Analogy: Think of it like a dating app for data. Instead of forcing every single person in a crowd to find a perfect match, the algorithm says, "Let's just find the best matches for the people who actually care about each other."
How it works: It uses a math trick called "Partial Optimal Transport" to align only the important parts of the video with the relevant parts of the text. It ignores the noise and focuses on the critical connections, ensuring the "sticky notes" learn the right things even when there isn't much data to learn from.

5. The Results: Small Size, Big Wins

The authors tested this system on two main tasks:

Finding the Moment (Temporal Language Grounding): Like finding the exact second in a video where "the girl smiles after the proposal."
Summarizing the Story (Video-Language Summarization): Like watching a video and writing a short summary of what happened.

The Outcome:

Efficiency: Their method only needed to train about 1.2% of the parameters (the "sticky notes") compared to the whole library.
Performance: Despite training so little, their method actually performed better than the massive, fully retrained libraries and other existing "lightweight" methods.
Versatility: It worked well on different types of video models and different datasets.

Summary

The paper proposes a way to teach giant AI models new video skills without breaking the bank or the storage. They do this by adding tiny, time-aware "sticky notes" (READ) that understand the flow of a story, and using a smart "matching game" (PVLA) that focuses only on the important connections between what is seen and what is said. The result is a system that is cheap to run, easy to store, and surprisingly accurate.

Technical Summary: READ-PVLA for Parameter-Efficient Video-Language Modeling

Problem Statement

Fully fine-tuning large-scale pre-trained transformer models for video-language tasks (such as Temporal Language Grounding and Video-Language Summarization) presents significant challenges, particularly in low-resource settings. This approach incurs high storage costs due to the need to store a separate model for each task and often leads to unstable training and sub-optimal performance when training data is limited.

While existing parameter-efficient methods, such as standard adapters, introduce lightweight modules to freeze the pre-trained backbone and only train the adapters, they exhibit two critical limitations:

Lack of Temporal Modeling: Standard adapters treat video frames and textual words as independent tokens, failing to capture intrinsic temporal dependencies among frames or sequential words. This hinders the model's ability to reason about temporal context (e.g., distinguishing events before or after a specific action).
Information Loss: Existing adapters often neglect the preservation of critical task-related information as it flows from raw inputs into the adapter's low-dimensional space. This can result in the loss of essential semantic information and the introduction of noise during the fine-tuning process.

Methodology

The authors propose the READ-PVLA framework, which addresses these issues through two novel components integrated into a parameter-efficient fine-tuning strategy.

1. Recurrent Adapter (READ)

To capture temporal dependencies, the authors introduce a Recurrent Adapter (READ). Unlike standard adapters that rely solely on non-linear projections, READ incorporates a recurrent bottleneck layer within the Transformer architecture.

Architecture: The module consists of a down-projection layer, a Recurrent Neural Network (RNN) layer, and an up-projection layer.
Operation: Given the output $O$ from a FeedForward block, the READ module computes a residual update: $\tilde{O} = O + \text{GELU}(\text{RNN}(O \cdot W_{down})) \cdot W_{up}$ .
Efficiency: By performing recurrent computations in a low-dimensional space (where the bottleneck dimension $k \ll d$ ), READ maintains a lightweight design, requiring only up to 1.20% of trainable parameters.
Flexibility: The framework is designed to be compatible with various pre-trained backbones, including Unified Multimodal Transformers (UMT), Moment-DETR, and generative vision-guided models like VG-BART and VG-T5.

2. Partial Video-Language Alignment (PVLA)

To ensure that critical information is preserved during the adaptation process, the authors propose a Partial Video-Language Alignment (PVLA) objective.

Concept: PVLA explicitly encourages alignment between video frame representations and textual word representations by minimizing the Partial Optimal Transport (POT) distance between their distributions.
Rationale: Video and language do not exhibit a complete one-to-one correspondence; language often describes only specific aspects of a video. Standard optimal transport would force a match between all masses, potentially aligning irrelevant information.
Mechanism: PVLA allows the algorithm to transport only a subset of samples ( $s$ ) from one distribution to the other, focusing on the "essential masses" that are strongly related. This is formulated as minimizing the cost $T_{i,j} \cdot c(h_{v,i}, h_{l,j})$ subject to partial transport constraints.
Training: During fine-tuning, the pre-trained backbone is frozen. The READ layers are optimized jointly using the task-specific loss and the PVLA loss.

Key Contributions

The paper outlines three primary contributions:

READ Architecture: A novel adapter design that integrates recurrent computation to effectively model temporal information in video-language tasks, overcoming the limitations of token-independent projections.
PVLA Objective: A new alignment objective using Partial Optimal Transport to maintain the flow of critical, task-relevant information between modalities during the bottleneck adaptation process.
Empirical Validation: Extensive experiments demonstrating that the READ framework outperforms both full fine-tuning and other parameter-efficient strategies (e.g., LoRA, Prompt, standard Adapters) on low-resource benchmarks, using significantly fewer trainable parameters (as low as 0.05% to 1.20%).

Experimental Results

The framework was evaluated on Temporal Language Grounding (TLG) and Video-Language Summarization (VLS) tasks across multiple datasets: YouTube Highlights, TVSum, QVHighlights, and How2.

Temporal Language Grounding (TLG):
- On YouTube Highlights, READ achieved a mean Average Precision (mAP) of 76.12%, outperforming full fine-tuning (74.76%) and the standard Adapter (68.98%), while using only 0.16M trainable parameters (0.06% of the total).
- On TVSum, READ achieved 83.39% mAP, surpassing full fine-tuning (79.65%) and the best parameter-efficient baseline (Adapter at 74.25%) with only 0.14M parameters.
- On QVHighlights, READ improved mAP to 36.74% over full fine-tuning (36.14%) using 0.19M parameters.
Video-Language Summarization (VLS):
- Using the VG-BART backbone on the How2 dataset, READ achieved ROUGE-1, ROUGE-2, and ROUGE-L scores of 44.01, 21.91, and 37.91, respectively. This significantly exceeded full fine-tuning (35.72, 11.88, 30.00) and other efficient methods, with only 1.17M parameters (0.47% of the model).
- Similar gains were observed with the VG-T5 backbone, where READ achieved 40.12 ROUGE-1, outperforming full fine-tuning (32.37) with 1.91M parameters.
Ablation Studies:
- Removing the PVLA objective caused a dramatic performance drop, confirming its role in preventing information loss and noise injection.
- The choice of recurrent architecture (RNN vs. GRU vs. LSTM) had minimal impact on performance, validating the use of the simplest RNN implementation.
- Partial Optimal Transport (POT) outperformed standard distance metrics (Cosine, L2) and full Optimal Transport, highlighting the importance of partial alignment.

Significance and Claims

The paper claims that the READ-PVLA framework offers a robust solution for parameter-efficient transfer learning in video-language modeling. By combining recurrent temporal modeling with a partial alignment objective, the method successfully addresses the instability and information loss associated with low-resource fine-tuning.

The authors emphasize that their approach:

Outperforms Full Fine-Tuning: It achieves higher accuracy than training all parameters, despite freezing the vast majority of the pre-trained model.
Superior Efficiency: It requires significantly fewer trainable parameters (up to 1.20%) compared to other efficient methods like LoRA or standard Adapters.
Generalizability: The framework is applicable to diverse pre-trained architectures, including both discriminative (UMT, Moment-DETR) and generative (VG-BART, VG-T5) models.

The work suggests that incorporating temporal dynamics and carefully managing information flow via partial alignment are critical for advancing video-language understanding in resource-constrained environments.