🤖 AI

T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs

This paper introduces T2T-VICL, a collaborative prompt-transfer framework that enables cross-task visual in-context learning by distilling implicit textual guidance from a teacher VLM into a lightweight student model, allowing it to effectively handle mismatched demonstration-query pairs across diverse vision tasks without explicit task naming.

Original authors: Shao-Jun Xia, Huixin Zhang, Zhengzhong Tu

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shao-Jun Xia, Huixin Zhang, Zhengzhong Tu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to fix things. Usually, you'd show it a picture of a muddy shoe and a clean shoe, and say, "See? This is how you clean shoes." If the robot then sees a muddy car, it might get confused: should it try to clean the car exactly like the shoe, or should it figure out that cars need a different kind of cleaning? This is the heart of a field called Visual In-Context Learning (VICL). It's a way for smart computer models to learn new jobs just by looking at a few examples, without needing to be retrained from scratch. Think of it like a super-smart apprentice who can pick up a new skill just by watching a master do it once. But here's the tricky part: most of these apprentices are trained to only copy what they see. If you show them a shoe-cleaning example but ask them to fix a car, they might try to scrub the car with a shoe brush, which doesn't work very well. The big question scientists are asking is: Can these AI apprentices learn to understand the idea of "cleaning" well enough to switch between totally different jobs, like cleaning a shoe versus cleaning a car, just by looking at the examples?

This paper introduces a clever new system called T2T-VICL (Task-to-Task Visual In-Context Learning) that tries to solve this exact problem. The researchers found that when you give a powerful AI a "mismatched" example—like showing it a rain-removal picture but asking it to remove fog—the AI often gets stuck trying to copy the rain removal instead of figuring out the fog removal. To fix this, the team built a two-step "teacher-student" team. First, a giant, super-smart AI (the teacher) looks at the mismatched pictures and writes a secret, invisible note describing how the two jobs are different, without ever naming the jobs themselves. It's like the teacher whispering, "Hey, the first picture needed water removed, but this new one needs air cleared, so be careful," without saying "rain" or "fog." Then, a smaller, faster AI (the student) learns to write these secret notes on its own. When a user asks the system to fix a new image, the student writes a custom instruction based on the example and the new image, and passes it to the final AI to do the actual work.

The results suggest that this method works surprisingly well. The team tested their system on 12 different low-level vision tasks, such as removing rain, haze, or noise, and even changing image styles. They tried over 20 different combinations where the example task and the target task were different. In many cases, using their "secret note" method improved the quality of the results compared to just giving the AI a fixed, generic instruction. For instance, when trying to turn a blurry image into a clear one and then asking the AI to remove haze, their method boosted the visual quality score (VIEScore) by as much as 2.24 points in some cases. The paper suggests that this approach helps the AI understand the relationship between tasks rather than just blindly copying pixels. However, the authors are careful to note that this isn't a magic bullet for every situation; sometimes the results were still a bit fuzzy, and the system works best when the tasks share some underlying similarities. They also point out that while this works great for fixing images, it might not be ready for much harder jobs like solving geometry problems yet. Ultimately, this research suggests that by translating visual differences into language, we can help AI models become more flexible and adaptable, bridging the gap between tasks that previously seemed too different to connect.

Technical Summary: T2T-VICL

Problem Statement
Visual In-Context Learning (VICL) enables foundation models to solve visual tasks using a few input-output demonstrations without parameter updates. While recent Large Vision-Language Models (VLMs) show promise in VICL when demonstrations and queries belong to the same task, real-world applications often present mismatched examples (e.g., a user providing a deraining demonstration for a dehazing query). Existing VICL methods largely rely on a restrictive "same-task" assumption. When demonstrations and queries differ, it is unclear whether a VLM should imitate the demonstrated transformation or infer a new one based on the query. This raises the fundamental question: Can VLMs perform cross-task VICL where the demonstration and query come from different tasks? Current fixed prompting strategies are insufficient for this setting as they provide generic instructions without resolving task discrepancies.

Methodology: T2T-VICL Framework
The authors propose T2T-VICL (Task-to-Task Visual In-Context Learning), a collaborative prompt-transfer framework designed to convert mismatched visual demonstrations into implicit textual guidance without explicitly naming the tasks. The pipeline consists of three main stages:

Implicit Task Relationship Generation (Teacher Phase):
- A large teacher VLM (Qwen2.5-VL-32B-Instruct) is used to generate structured, implicit textual descriptions of the differences between two distinct low-level vision tasks (Task A and Task B).
- The input consists of an image pair from Task A (input and label) and an image pair from Task B (input and label).
- The prompt instructs the model to describe: (1) image descriptions, (2) visual changes, and (3) task differences, without explicitly naming the tasks. This forces the model to articulate subtle visual changes in a narrative, task-agnostic manner.
- A diversity filtering mechanism using semantic sentence embeddings (SentenceTransformer) removes repetitive outputs, resulting in a benchmark dataset of 2,000 diverse implicit cross-task relations per task pair.
VLM-sVLM Knowledge Transfer (Distillation Phase):
- A lightweight student VLM (sVLM, Qwen3-VL-4B-Instruct) is fine-tuned to mimic the teacher's reasoning.
- The student is trained to generate the implicit comparison text ( $P'_r$ ) given the input images from Task A and the input image from Task B, using an open-ended prompt. Crucially, the student does not receive the label of Task B during inference training, forcing it to learn the task relationship solely from visual characteristics.
- This transfers the "reasoning habit" of the large model into a smaller, efficient model capable of generating dynamic, content-dependent prompts.
Score-Based Inference Framework (Deployment Phase):
- During inference, the trained sVLM processes the demonstration pair (Task A) and the query image (Task B) to generate an implicit text prompt ( $P''_t$ ).
- This prompt is fed into a frozen, larger image-editing VLM (e.g., Gemini-2.5-Flash or Seedream 4.0) to guide the final image generation.
- To handle the stochastic nature of generation, a score-based inference strategy is employed. Multiple candidate outputs are generated and ranked using a hybrid metric suite:
  - VIEScore: A VLM-based evaluator that assesses Semantic Consistency (SC) and Perceptual Quality (PQ) via natural language rationales.
  - Traditional Metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM).
- The candidate with the highest score is selected as the final output.

Key Contributions

Problem Formulation & Benchmark: The paper formulates cross-task VICL for low-level vision and introduces a benchmark dataset containing implicit cross-task descriptions, enabling systematic study of VLM generalization across mismatched visual demonstrations.
T2T-VICL Pipeline: A novel VLM $\to$ sVLM $\to$ VLM teacher-student pipeline that distills implicit task-relation knowledge from a large model into a lightweight prompt generator, allowing frozen image-editing VLMs to adapt to cross-task scenarios without retraining.
Score-Based Inference: A framework coupling attribution-enhanced prompting, visual instruction scoring (VIEScore), and image quality assessment (IQA) metrics to automatically support and evaluate cross-task VICL.

Results
Experiments were conducted on 12 low-level vision tasks (including restoration, removal, and generation/enhancement tasks) covering over 20 evaluated cross-task pairs.

Performance: T2T-VICL consistently improves task-aware alignment (measured by VIEScore) over fixed prompting baselines across both Gemini and Seedream models.
Fidelity: In many cases, the method also improves image fidelity (PSNR and SSIM), particularly in the "top-tier" task pairs. In "second-tier" pairs, improvements are primarily reflected in VIEScore (semantic/structural coherence) rather than strict pixel-level fidelity.
Generalization: The framework demonstrates robustness in generalizing to semantically distant task pairs (e.g., Denoising $\to$ Light Enhancement) where specialized models typically degrade without retraining.
Qualitative: Visual examples show the model successfully adapting to diverse tasks (e.g., using a deraining example to perform dehazing) while maintaining spatial layout and identity fidelity.

Significance and Claims
The paper claims to present the first cross-task visual in-context learning pipeline that enables collaboration among multiple VLMs with an automatic reasoning mechanism.

Significance: It systematically explores the "uncharted boundaries" of cross-task transfer in VICL, demonstrating that language can serve as a natural bridge for transferring knowledge across mismatched visual tasks.
Modesty: The authors acknowledge limitations, noting that the scope is currently restricted to low-level vision tasks. They state that task transfer capability is not unlimited and depends on the underlying similarity of tasks. They also note that in some cases with limited reference samples, the method may produce content not present in original images, slightly lowering fidelity-based indicators.
Future Outlook: The work suggests that understanding how heterogeneous VLMs jointly reason and adapt across tasks is a critical step toward broader generalization capacity in multimodal models.

Technical Summary: T2T-VICL

More like this