DuoTeach: Dual Role Self-Teaching for Coarse-to-Fine Decision Coordination in Vision--Language Models

1. Problem Statement

The paper addresses the challenge of coarse-to-fine path decision-making in Vision-Language Models (VLMs). In many applications (e.g., product categorization, medical coding), a model must not only predict a fine-grained label but also generate a valid ancestry path through a taxonomy (e.g., Animalia $\to$ Chordata $\to$ Mammalia $\to$ Carnivora).

Key Issues Identified:

• Evaluation Mismatch: Existing benchmarks typically evaluate each level of the hierarchy independently (level-wise scoring). This fails to capture the requirement that lower-level decisions must be consistent with higher-level decisions made within the same inference call.
• The "Joint Path" Failure: When strong VLMs are forced to predict the full path in a single call (Joint Path Decision), they frequently produce invalid parent-child pairs (e.g., predicting "Animalia" then "Arthropoda" then "Mammalia").
• Root Cause: The authors argue that these failures are not solely due to a lack of taxonomic knowledge but stem from unstable cross-level decision coordination. The model struggles to condition subsequent decisions on its own prior outputs within a single generation sequence.

2. Methodology: DuoTeach Framework

The authors propose DuoTeach, a dual-role self-teaching distillation framework designed to improve cross-level coordination without requiring ground-truth labels or external teacher models.

A. New Evaluation Protocol: JPD & DWPA

To properly evaluate this setting, the paper introduces:

• Joint Path Decision (JPD) Protocol: The model receives the image and all level-specific queries simultaneously and must output the full path in one call.
• Depth-Weighted Prefix Accuracy (DWPA): A new metric family that measures path reliability. It weights correctness based on depth, ensuring that a correct leaf node is only credited if all its ancestors are also correct.
  • $DWPA_{0.95}$: Emphasizes strict correctness at the finest (leaf) level.
  • $DWPA_{1/L}$: Measures the length of the longest correct prefix.

B. The DuoTeach Training Pipeline

DuoTeach reuses a single frozen pretrained VLM in two distinct roles:

1. Teacher Role (Decision-Conditioned Rollout - DCR):

   • The frozen teacher performs a compute-intensive, multi-step rollout.
   • It answers level 1, appends the decision to the prompt, answers level 2 based on that context, and so on.
   • This generates coherent teacher traces where decisions are explicitly conditioned on prior outputs, minimizing parent-child conflicts.
   • The teacher outputs: hard decisions, probability distributions, and hidden states at "anchor points" (the token representing the decision).

2. Student Role (One-Call Inference):

   • A trainable student model (initialized from the same VLM) is trained to perform the JPD task (predicting the full path in a single call).
   • The student receives the image and all queries at once, without injected prior decisions.

3. Distillation Objective:
   The student is trained to align with the teacher's traces at specific decision points using a composite loss function:

   • Hard Loss ($\mathcal{L}_{hard}$): Cross-entropy between student predictions and teacher's hard decisions.
   • Soft Loss ($\mathcal{L}_{soft}$): KL-divergence between student and teacher probability distributions.
   • Feature Loss ($\mathcal{L}_{feat}$): MSE loss aligning the student's hidden states (at the decision anchor) with the teacher's hidden states.

This process distills the "reasoning process" (decision conditioning) from the multi-step teacher into the efficient one-call student.

3. Key Contributions

1. Protocol & Metric: Introduction of the JPD protocol and DWPA metric, shifting the evaluation focus from isolated level accuracy to coherent full-path reliability.
2. Diagnosis of Failure: Demonstration that strong VLMs suffer from cross-level decision coordination bottlenecks under JPD, leading to inconsistent paths even when individual level accuracy is high.
3. DuoTeach Framework: A novel self-teaching distillation method that:
   • Requires no ground-truth labels (uses self-generated traces).
   • Requires no external teacher models (uses the same frozen VLM).
   • Eliminates inference overhead (student runs in one call, unlike the multi-step teacher).
4. Generalization: Proof that the learned coordination mechanism transfers to unseen taxonomies and general compositional reasoning tasks.

4. Experimental Results

The method was evaluated on multiple taxonomy-structured datasets (iNaturalist, ImageNet, CUB-200, Food-101) and various VLM backbones (LLaVA-OV, InternVL, Qwen2.5-VL).

• Performance Gains:

  • On iNaturalist-Animal, DuoTeach improved $DWPA_{0.95}$ by +30.24 points for LLaVA-OV-7B (from 0.69% to 30.93%).
  • It significantly boosted zero-shot performance on unseen taxonomies (Food-101), increasing $DWPA_{0.95}$ from 17.17% to 43.66% for Qwen2.5-VL-7B.
  • Consistent improvements were observed across all metrics ($DWPA$, LeafAcc, TOR) and model sizes.

• Ablation & Analysis:

  • Coordination vs. Knowledge: Comparisons with Supervised Fine-Tuning (SFT) showed that DuoTeach outperforms SFT on unseen taxonomies, proving it learns a general coordination strategy rather than just memorizing specific labels.
  • Loss Components: Removing any of the three loss terms (Hard, Soft, Feature) degraded performance, confirming the necessity of aligning both decisions and internal states.
  • Efficiency: The 7B DuoTeach student achieved competitive results against a 32B Base model while using 4.0x less VRAM and achieving 3.34x lower latency (single call vs. 7-step rollout).

5. Significance

• Paradigm Shift: The paper highlights a critical flaw in current hierarchical evaluation methods (level-wise scoring) and proposes a more rigorous standard (JPD) that reflects real-world deployment constraints.
• Efficiency vs. Reasoning: It demonstrates that complex, multi-step reasoning (typically requiring expensive inference rollouts) can be distilled into efficient, single-pass inference without sacrificing coherence.
• Self-Improvement: By leveraging self-teaching without external labels, the framework offers a scalable solution for improving VLMs in structured decision-making tasks where ground-truth paths are scarce or expensive to annotate.
• Broad Applicability: The success on unseen taxonomies and external benchmarks (GQA, MathVista) suggests the method improves general multi-step reasoning and state maintenance capabilities in VLMs, beyond just taxonomy prediction.