Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model
Team Mia won the TextVQA Challenge 2021 by fine-tuning a pre-trained T5-3B generative model using specialized pre-training tasks—masked language modeling and relative position prediction—to effectively align multi-modal features for reading and reasoning about text in images.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are playing a high-stakes game of "I Spy," but instead of just looking for colors or shapes, you have to read tiny, blurry signs, menus, and labels in a crowded photograph to answer complex questions.
For example, if someone shows you a photo of a busy cafe and asks, "What is the name of the drink on the third table from the left?" you can't just see a "cup"—you have to zoom in, read the text on the cup, and connect it to the location.
This is the TextVQA Challenge, and the paper describes how Team Mia built a "super-brain" to win this game. Here is how they did it, broken down into simple ideas.
1. The Brain: The "Master Translator" (T5 Model)
Most AI models are good at either seeing (Vision) or reading (Language), but they often struggle to do both at once. Team Mia used a model called T5, which is like a world-class translator.
Instead of just labeling an image as "a kitchen," this model is designed to take a messy mix of information—the question, the objects it sees, and the words it reads—and "translate" that mess into a clear, spoken answer.
2. The Training: "The Extreme Library Study Session"
Before the model ever saw the actual competition questions, the team put it through a massive "study session."
- The Massive Library: They didn't just give it a few books; they gave it 9 million images containing text. It’s like forcing a student to read 9 million street signs, labels, and posters before the exam starts.
- The Two Special Drills: To make sure the brain was sharp, they gave it two specific types of homework:
- The "Fill-in-the-Blanks" Drill (MLM): They would hide a word in a sentence and make the model guess it. This teaches the model how language flows.
- The "Where is it?" Drill (RPP): They would show the model a piece of text and an object and ask, "How far apart are these?" This teaches the model to connect the word "Coca-Cola" to the actual red can sitting on the table.
3. The Secret Sauce: "Adversarial Training"
Imagine you are practicing basketball. If you only practice shooting against a hoop that never moves, you’ll be terrible in a real game where defenders are jumping in your face.
The team used Adversarial Training. They essentially "poked" and "nudged" the model's internal logic during training, making it harder for the model to get the right answer easily. This forced the AI to become "tougher" and more stable, so it wouldn't get confused by blurry text or weird lighting in the real competition.
4. The Final Polish: "The Spell-Checker"
Even the smartest student might make a tiny typo, like writing "Appel" instead of "Apple." To prevent this, the team added a Post-processing Module (using a tool called fuzzywuzzy).
Think of this as a friendly editor sitting next to the AI. If the AI outputs an answer that is almost right but has a tiny spelling error, the editor steps in and says, "I think you meant 'Apple,' right?" and corrects it before submitting the answer.
Summary: The Winning Recipe
To win, Team Mia followed a simple logic:
- Start with a genius: Use a massive, pre-trained language model.
- Feed it everything: Give it millions of images so it knows what the world looks like.
- Teach it connections: Make sure it knows that a word and an object are actually the same thing.
- Make it tough: Train it against "distractions" so it doesn't get confused.
- Double-check the work: Use a spell-checker to clean up the final answer.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.