💬 NLP

Retrieval-Augmented Long-Context Translation for Cultural Image Captioning: Gators submission for AmericasNLP 2026 shared task

The University of Florida Gators won the AmericasNLP 2026 shared task on cultural image captioning for Indigenous languages by employing a two-stage pipeline that combines Qwen2.5-VL for Spanish intermediate captioning with retrieval-augmented many-shot prompting using Gemini 2.5 Flash, achieving over 120% performance improvements over the baseline across Bribri, Guaraní, and Orizaba Nahuatl.

Original authors: Aashish Dhawan, Christopher Driggers-Ellis, Dzmitry Kasinets, Daisy Zhe Wang, Christan Grant

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aashish Dhawan, Christopher Driggers-Ellis, Dzmitry Kasinets, Daisy Zhe Wang, Christan Grant

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a short, culturally perfect description of a photo for a community that speaks a rare, ancient language. The challenge is that you don't speak that language well, and there are very few books or dictionaries available to help you.

This paper describes how a team from the University of Florida (the "Gators") solved this problem for a competition called AmericasNLP 2026. Their goal was to take a picture and write a caption in one of five Indigenous languages of the Americas (like Guaraní or Bribri).

Here is how they did it, broken down into simple steps and analogies:

The Two-Step Recipe

Instead of trying to jump straight from "Picture" to "Rare Language," the team used a two-step relay race:

Step 1: The Translator (The Bridge): First, they used a super-smart AI (called Qwen) to look at the picture and write a simple description in Spanish. Think of Spanish as a "bridge language" that the AI is very good at.
Step 2: The Cultural Expert (The Finisher): Next, they took that Spanish description and asked a different, even smarter AI (Gemini) to translate it into the final Indigenous language.

The Secret Sauce: "Show, Don't Just Tell"

The real magic wasn't just in the translation; it was in how they taught the AI what "style" to use.

Usually, when you ask an AI to translate, it might give you a sentence that is grammatically correct but sounds like a robot or a textbook. The team wanted the captions to sound like a local person speaking naturally.

To fix this, they used a technique called Retrieval-Augmented Generation.

The Analogy: Imagine you are writing a letter to a friend in a foreign village. Instead of just guessing how they speak, you pull out a box of 100 letters that other people have written to that same village. You read them to get a feel for the slang, the tone, and the sentence structure. Then, you write your letter, mimicking that style.
In the Paper: Before the AI translated the Spanish caption, the system searched a massive library of existing Spanish-to-Indigenous-language pairs. It grabbed the most similar examples (like the "100 letters") and showed them to the AI as a guide. This helped the AI match the "register" (the specific cultural style) required by the competition.

The Results: A Big Win

The team entered this system into the competition and won.

The Score: They improved their scores by huge margins compared to the standard baseline. For example, in the Bribri language, they improved the score by 164%. In Guaraní, they improved by 131%.
The Human Test: When human judges looked at the captions, the team's entries were ranked second out of five finalists, proving the captions sounded natural and culturally appropriate.

What They Learned (The "Aha!" Moments)

The team ran many experiments to see what worked and what didn't, leading to three key discoveries:

Not All Libraries Are Equal: The "library" of examples they used worked wonders for Guaraní because they had a huge collection of 53,000 examples (including some computer-generated "fake" examples that turned out to be very helpful). However, for Yucatec Maya, they had almost no examples. In that case, the AI's own internal knowledge was better than trying to force it to use a tiny, noisy library.
The "Fake" Data Boost: For Guaraní, about 28 points of their success came specifically from using those computer-generated examples. It's like having a practice coach that creates extra drills for you to study.
Tone Matters (Especially for Bribri): The Bribri language has complex sounds (tones) that change the meaning of words. The team found that simply translating wasn't enough; they had to give the AI special instructions on how to handle these sounds and word order. It was like telling the AI, "Remember, in this language, the verb goes at the end, and don't forget the musical notes!"

The Catch (Limitations)

The team admits their system isn't perfect yet:

The Chain Reaction: If the first AI (the Spanish translator) makes a mistake describing the picture, the second AI (the translator) will translate that mistake perfectly. There is no "undo" button.
The Test Score Trap: They used examples from the "practice test" (development set) to help the AI learn the style. This worked great for the practice scores, but it's a bit like studying the exact answers to a practice quiz before taking the real exam. It might inflate the score, so they are careful about how they interpret those results.

Summary

The University of Florida team won by building a pipeline that first describes a picture in Spanish, then uses a massive library of similar examples to teach an AI how to translate that description into Indigenous languages with the right cultural flavor. They proved that for low-resource languages, context and style guides are just as important as the translation itself.

Technical Summary: Retrieval-Augmented Long-Context Translation for Cultural Image Captioning

Problem Statement
The paper addresses the AmericasNLP 2026 shared task, which requires generating culturally grounded image captions in five Indigenous languages of the Americas: Guaraní, Yucatec Maya, Orizaba Nahuatl, Bribri, and Wixárika. The authors identify three primary challenges: the target languages are low-resource; the captions must be culturally specific rather than generic visual descriptions; and the system must achieve precise stylistic control to match a specific register (short, natural sentences). The authors note that standard Vision-Language Models (VLMs) fail to generate captions directly in these target languages due to limited training data, and standard Neural Machine Translation (NMT) pipelines (specifically mBART-based systems) fail to capture the required cultural register, often producing fluent but stylistically mismatched outputs.

Methodology
The University of Florida Gators team proposes a two-stage pipeline that shifts from sequence-to-sequence translation to retrieval-augmented in-context learning using Large Language Models (LLMs).

Stage 1: Intermediate Captioning: A VLM (either Qwen2.5-VL-72B or Qwen3-VL-8B) generates a culturally aware, noun-first Spanish caption for each input image. This stage is treated as a fixed component.
Stage 2: Retrieval-Augmented Translation: The Spanish caption serves as a query for a BM25 retrieval system over a language-specific Spanish-to-Target-Language (Es-LoRes) parallel corpus.
- Prompt Construction: The final prompt is a "many-shot" context containing:
  - Retrieved Training Examples ( $R_r$ ): $r$ pairs of Spanish and target-language text retrieved based on lexical similarity to the query.
  - Development Exemplars ( $D_d$ ): $d$ gold-standard examples from the shared task's development set to guide the stylistic register.
- Generation: An LLM (primarily Gemini 2.5 Flash) translates the Spanish caption into the target language, conditioned on the retrieved and development examples. The system uses a temperature of 0.0 and disables "thinking" modes to ensure concise output.
- Language-Specific Adaptations:
  - Guaraní: Utilizes a large retrieval bank (53k+ pairs) augmented with synthetic data from the MultiScript30k project.
  - Yucatec Maya: No external retrieval bank exists; the system relies solely on development exemplars and the LLM's pre-trained knowledge ( $r=0$ ).
  - Bribri: Requires specific morphological post-processing (NFD normalization) to handle tonal diacritics correctly and specialized prompting for Subject-Object-Verb (SOV) word order and possessive prefixes.

Key Contributions

Pipeline Architecture: The authors present a retrieval-augmented LLM pipeline that outperforms traditional fine-tuned NMT baselines for low-resource cultural captioning.
Extensive Ablation Studies: The paper documents a comprehensive set of experiments varying model choices (OpenAI vs. Gemini), retrieval sizes ( $r$ ), development exemplar counts ( $d$ ), and prompting strategies.
Evaluation of Synthetic Data: The study isolates the impact of synthetic data augmentation, finding it to be a critical factor for Guaraní performance.
Methodological Insights: The authors highlight that retrieval effectiveness is highly language-dependent and that using development set examples as in-context prompts can inflate development scores, requiring careful interpretation.

Results
The system achieved the overall winner status for the shared task, placing second in human evaluations of the target-language captions.

Performance Gains: The system achieved significant improvements over the shared task baseline (Gow-Smith and Sánchez Villegas, 2023) in the development set:
- Bribri: 164.1% improvement.
- Guaraní: 131.7% improvement.
- Orizaba Nahuatl: 122.6% improvement.
- Wixárika: 6.87% improvement.
Test Set Performance: Improvements were maintained for Bribri (155.2%) and Orizaba Nahuatl (166.9%) in the test set. However, Guaraní performance dropped significantly in the test evaluation, with the authors noting that the development set gains were not fully reproduced.
Model Comparison: Gemini 2.5 Flash demonstrated dramatically superior performance in Es-LoRes in-context translation compared to OpenAI models (GPT-4o-mini, GPT-4.1) and the mBART baseline.
Synthetic Data Impact: Ablation studies revealed that synthetic exemplars from MultiScript30k accounted for approximately 28 chrF++ points of the Guaraní performance gain. Without synthetic data, Guaraní performance dropped by over 50% relative to the best synthetic-augmented result.
Morphological Handling: For Bribri, the combination of NFD normalization and morphological prompting contributed over 10% of the total performance gain over the baseline.

Significance and Claims
The paper claims that retrieval-augmented in-context learning is a viable and superior alternative to fine-tuned NMT for low-resource cultural image captioning, provided that the retrieval corpus is sufficiently large and in-domain. The authors emphasize that:

Retrieval is Language-Dependent: Large retrieval windows benefit languages with large, aligned corpora (like Guaraní) but can introduce noise for languages with sparse data (like Yucatec Maya).
Synthetic Data is a Driver: Synthetic parallel data significantly boosts performance for languages where such augmentation is available.
Domain Mismatch is a Bottleneck: For the lowest-resource languages (Wixárika and Bribri), the primary limitation is the mismatch between available corpus domains (narrative/literary) and the visual captioning task, rather than model capacity.
Evaluation Caution: The use of development set examples as in-context prompts is a double-edged sword; while it helps match the required register, it risks inflating development scores if the same pool is used for both selection and evaluation.

The authors conclude that while their system is a cascade (where errors in the Spanish caption propagate), the translation stage's ability to adapt to style and vocabulary via retrieval is the primary source of improvement over existing baselines. They suggest future work should focus on improving the initial visual captioning stage and acquiring domain-specific parallel data for the lowest-resource languages.