TPCL: Task Progressive Curriculum Learning for Robust Visual Question Answering

The paper introduces Task-Progressive Curriculum Learning (TPCL), a model-agnostic framework that enhances Visual Question Answering robustness across in-distribution, out-of-distribution, and low-data settings by progressively training models on questions ordered by semantic type and Optimal Transport-based difficulty, achieving state-of-the-art performance without relying on data augmentation or explicit debiasing.

The Gist

Imagine you are trying to teach a robot to answer questions about pictures. This is called Visual Question Answering (VQA).

Right now, most robots are like students who are great at memorizing answers for a specific test but fail miserably when the questions change slightly. If they see a picture of a dog and the question is "Is this a dog?", they might guess "Yes" just because 90% of the pictures in their training book had dogs. They aren't actually looking at the picture; they are just guessing based on patterns. This is called "bias," and it makes them brittle.

The authors of this paper, Ahmed Akl and his team, have come up with a new way to train these robots called TPCL (Task-Progressive Curriculum Learning).

Here is the simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "Cramming" Student

Imagine a student who is forced to study for a math test by reading every single problem in the textbook in random order.

• They might get stuck on the hardest calculus problems on day one and give up.
• Or, they might memorize the answers to the easy questions but never learn the logic behind the hard ones.
• When the teacher gives them a new type of problem (one they haven't seen before), they panic because they only memorized the specific examples, not the underlying rules.

Current AI models do exactly this. They see all the data at once, get confused by the easy stuff, and fail when the data changes.

2. The Solution: The "Smart Syllabus" (Curriculum Learning)

The authors realized that humans don't learn by doing everything at once. We learn in a curriculum:

1. We learn to add single digits first.
2. Then we learn multiplication.
3. Then we learn algebra.
4. Finally, we tackle calculus.

The paper proposes doing the same for AI, but with a twist. Instead of just ordering questions by "easy to hard," they group questions by type (like "Yes/No" questions, "How many" questions, or "What color" questions) and then order those groups.

3. The Secret Sauce: The "Optimal Transport" Compass

How do you know which group of questions is harder?

• Old way: You might just count how many questions the robot gets wrong on average. But that's like saying "This student is bad at math because they got 50% of the questions wrong," without realizing they got the easy ones right and the impossible ones wrong.
• The new way (TPCL): The authors use a mathematical tool called Optimal Transport.

The Analogy: Imagine you have a pile of sand (the robot's mistakes) and you want to move it to a new spot.

• If the pile of sand is small and compact, it's easy to move (the task is easy).
• If the pile is scattered all over the floor, it takes a lot of effort to gather it up (the task is hard).

TPCL watches how the robot's "pile of mistakes" shifts and changes shape over time. If the shape of the mistakes changes wildly, the robot is struggling with that specific type of question. If the shape stays stable, the robot has mastered it.

4. The Training Strategy: "Hard First, Then Easy"

Here is the most surprising part. Most people think you should start with easy things. But TPCL does the opposite for the types of questions:

1. Start with the hardest question types (the ones the robot struggles with the most).
2. Force the robot to focus on these difficult patterns first.
3. Once the robot has "toughened up" and learned the hard logic, it gradually introduces the easier questions.

Why? Think of it like training for a marathon. If you start by running on a flat, easy track, you might get comfortable and lazy. But if you start by running up a steep hill, your legs get strong. Once you can run up the hill, the flat track feels like a breeze.

By forcing the robot to tackle the "steep hills" (hard question types) first, it learns to actually look at the picture rather than just guessing.

5. The Result: A Super-Adaptable Robot

Because the robot learned the hard logic first, it doesn't get confused when the test changes.

• Before: The robot was like a parrot that only repeats what it heard in the classroom.
• After (with TPCL): The robot is like a detective who understands the clues.

The paper shows that this method works incredibly well. The robot became much better at answering questions about pictures it had never seen before (Out-of-Distribution), and it didn't need any extra data or complex tricks to do it. It just needed a better syllabus.

Summary

• The Issue: AI models are too lazy; they guess based on patterns instead of looking at the image.
• The Fix: Don't feed them random data. Feed them a structured plan (Curriculum).
• The Method: Group questions by type, measure how hard each group is using a "sand-moving" math trick, and start training with the hardest groups first.
• The Outcome: A robot that is robust, smart, and can handle new situations without breaking a sweat.

Technical Summary

1. Problem Statement

Visual Question Answering (VQA) models suffer from a lack of robustness, particularly when facing distribution shifts (Out-of-Distribution or OOD scenarios) and data scarcity.

• The Core Issue: Current VQA models rely heavily on dataset biases (e.g., language priors where the answer is guessed based on the question alone) rather than genuine image understanding. While they perform well on In-Distribution (IID) data, they fail significantly on OOD datasets (like VQA-CP) where answer distributions differ from training data.
• Limitations of Existing Solutions:
  • Data Augmentation/Ensemble Methods: Strategies like Counterfactual Sample Synthesis (CSS) or ensemble learning (e.g., GenB, RUBi) often require additional annotations, can introduce semantic errors, or are sensitive to specific backbone architectures.
  • Training Strategy Flaw: Most existing methods treat all training samples uniformly, ignoring the varying linguistic difficulty and semantic structure of different question types. This prevents models from learning robust representations that generalize across different distributions.

2. Methodology: Task Progressive Curriculum Learning (TPCL)

The authors propose TPCL, a model-agnostic training framework that reframes VQA as a Multi-Task Learning (MTL) problem. Instead of training on a flat dataset, TPCL organizes learning into a curriculum of tasks based on question types.

A. Task Decomposition

The dataset is decomposed into $T$ sub-tasks based on question types (e.g., "Yes/No", "Counting", "Wh-questions", "Others"). In the VQA-CP context, there are 65 fine-grained question types.

• Key Insight: The paper draws on psycholinguistics, noting that certain question types (like "Wh-") are generally easier to acquire than others (like binary "Yes/No" in specific contexts), suggesting a structured learning order improves generalization.

B. The Two Core Components

TPCL operates using two integrated components: a Difficulty Measurer and a Pacing Function.

1. Difficulty Measurer (Novel Contribution)
Unlike previous Curriculum Learning (CL) works that measure difficulty per sample (instance-based), TPCL measures difficulty per task (task-based).

• Distributional Difficulty: Instead of averaging loss, TPCL tracks the distribution of loss scores for all samples within a specific question type.
• Optimal Transport (OT): To measure the difficulty of a task, the method calculates the divergence between the loss distributions of the current iteration and the previous one using Wasserstein Optimal Transport.
  • Rationale: As training progresses, loss distributions shift horizontally toward zero. Standard metrics like KL-divergence fail when distributions do not overlap. OT accounts for the underlying geometry of the distributions, making it robust to these shifts.
  • Interpretation: Tasks with high distributional divergence (unstable loss) are considered "harder" and are prioritized. Tasks with low divergence are "easier."
• Consolidation: To avoid noise from instantaneous hardness, the difficulty score is consolidated over a window of $B$ iterations using a weighted sum, prioritizing recent model states.

2. Pacing Function
This function controls the rate at which tasks are introduced to the model.

• Dynamic Variant (TPCLDyn): The model starts with the most difficult tasks (based on the OT metric) and progressively introduces easier tasks. This is an "anti-curriculum" approach (Hard $\to$ Easy), which the authors found superior for OOD generalization.
• Fixed Variant (TPCLFix): Tasks are ordered offline based on psycholinguistic heuristics (e.g., Binary $\to$ Number $\to$ Others $\to$ Wh-).
• Pacing Schedule: A step function gradually increases the fraction of the dataset (tasks) exposed to the model over training iterations.

3. Key Contributions

1. Task-Based Curriculum in VQA: First introduction of Curriculum Learning in VQA where the atomic unit is the task (question type) rather than the individual sample.
2. Optimal Transport Difficulty Metric: A novel, self-taught difficulty measure that uses the divergence of loss distributions (via Optimal Transport) to dynamically rank tasks, avoiding the pitfalls of mean-based metrics.
3. Robustness without Augmentation: The method achieves state-of-the-art robustness without requiring data augmentation, explicit debiasing branches, or additional annotations. It relies solely on the training strategy.
4. Generalization: Demonstrates that learning difficult tasks first (Hard-to-Easy) significantly improves OOD generalization compared to standard training or Easy-to-Hard approaches.

4. Experimental Results

The method was evaluated on VQA-CP v2, VQA-CP v1 (OOD benchmarks), and VQA v2 (IID benchmark) using backbones like LXMERT, UpDn, and SAN.

• State-of-the-Art Performance:
  • VQA-CP v2: TPCL (Dynamic) achieved 77.23% overall accuracy, outperforming the previous best (FAN-VQA) by 5.05%.
  • VQA-CP v1: Achieved 76.15%, outperforming the competitive baseline by 6.68%.
  • VQA v2 (IID): Achieved 78.03%, showing that robustness does not come at the cost of in-distribution performance.
• Backbone Agnostic: The framework improved performance across all tested backbones (LXMERT, SAN, UpDn), with gains up to 28.5% on LXMERT for OOD tasks.
• Low-Data Regime: In experiments using only 30% of the data, TPCL (Hard-to-Easy) achieved SOTA performance (72.58%), significantly outperforming forward (Easy-to-Hard) curriculum learning.
• Ablation Studies:
  • OT vs. Mean: Using Optimal Transport for difficulty measurement yielded ~1.37% to 2% higher accuracy compared to simple mean-loss metrics.
  • Dynamic vs. Fixed: Dynamic difficulty (Hard-to-Easy) generally outperformed fixed heuristics in OOD settings, though Fixed was still strong.

5. Significance and Conclusion

The paper establishes that training strategy is as critical as model architecture or data augmentation for robust VQA.

• Paradigm Shift: It challenges the standard "random sampling" or "uniform training" approach, proving that structuring the learning process based on semantic task difficulty leads to better feature learning.
• Efficiency: TPCL offers a computationally efficient solution (negligible overhead for OT calculation) that eliminates the need for complex debiasing modules or synthetic data generation.
• Generalizability: The findings suggest that "Hard-to-Easy" curriculum learning is a powerful mechanism for overcoming dataset biases and achieving true out-of-distribution generalization in multi-modal tasks.

The source code is publicly available, facilitating further research into curriculum-based robust learning.