FAU at ImageCLEF 2026 Task on Multimodal Reasoning Robust Candidate Scoring and Concise Multilingual Visual Answering

Imagine you are sitting in a high-stakes exam hall, but instead of just reading words on a page, the test is a chaotic collage of dense textbooks, complex diagrams, scribbled formulas, and charts in six different languages. Your job is to look at this visual mess and answer tricky questions about it. This is the world of "Multimodal Reasoning," a field where computers try to act like human students who can both "see" an image and "read" the text inside it. For a long time, the big question was: "How smart does the computer's brain need to be to pass?" But there's a catch. Even if a super-smart computer understands the math and the science perfectly, it can still fail the test if it talks too much, uses the wrong language, or writes its answer in a messy format that the teacher's grading machine can't read. It's like having a genius student who writes a brilliant essay but forgets to write their name on the line, so they get a zero.

This paper tells the story of a team from Germany (FAU) who entered a competition called ImageCLEF 2026 to solve exactly this problem. They didn't just build a smarter brain; they built a stricter teacher. Their main discovery is that how you ask the computer to answer is just as important as the computer itself. They found that for multiple-choice questions, it's better to stop the computer from "writing" an answer and instead have it "score" the options like a judge. For open-ended questions, they learned that the computer needs to be forced to be brief and clean, stripping away all its "thinking out loud" before submitting the final result. They also tried some popular tricks, like giving the computer a reference sheet of text (OCR) or teaching it new things (fine-tuning), but surprisingly, those tricks actually made the computer perform worse. By focusing on strict rules and cleaning up the answers, their system climbed to the top of the leaderboard, proving that a little bit of engineering discipline can turn a powerful but messy AI into a reliable test-taker.

The Challenge: The Visual Exam

The competition had two main types of questions. The first was Visual Multiple Choice (Visual MCQ). Imagine a biology question in Bulgarian with a table of data and five options labeled A through E. The computer has to pick the right letter. The second was Visual Open Question Answering (Visual OpenQA). Here, there are no options. The computer has to look at a diagram (maybe an economics chart) and write a short, direct answer in the same language as the question.

The tricky part is that these images are full of "dense" information. They aren't just pictures of cats; they are pages of scientific papers with formulas, units, and tiny text. If the computer gets confused by the layout or writes a long explanation when only a number is needed, it loses points.

The Strategy: The "Score, Don't Write" Trick

For the multiple-choice part, the team realized that letting the computer write a sentence like "The answer is A because..." was dangerous. The computer might get the right letter but wrap it in too much extra text, confusing the grading system.

Instead, they changed the game. They asked the computer to look at the options A, B, C, D, and E and simply give each one a "score" based on how likely it was to be correct. Think of it like a sports judge giving a score of 9.5 to a gymnast's routine instead of writing a paragraph about it. The computer didn't have to generate text; it just had to calculate which letter had the highest score. This made the answers incredibly clean and easy to combine. If they ran the test with three different computer brains, they could take the scores from all three, average them out, and pick the winner. This "score fusion" helped them get a very high accuracy of 0.7108, landing them in third place overall.

The Cleanup Crew: Taming the Chatty AI

For the open-ended questions, the problem was the opposite. The computers were too chatty. They loved to explain their reasoning, say "Here is the answer," or use fancy XML tags like <answer>. The grading system, however, just wanted the raw answer.

The team treated this like a strict editor. They used a "concise prompt" that told the computer: "Stop thinking out loud. Just give me the answer." Then, they ran the answers through a "cleanup crew" (a set of rules) that deleted any extra words, reasoning traces, or formatting errors. They also combined the answers from several different computer models. If one model said "50 kg" and another said "50 kilograms," the system figured out they were the same and picked the best one. This careful cleaning and combining helped them win first place in the open-ended category with a score of 0.6488 (measured by a metric called COMET).

The Surprising "Don'ts": What Didn't Work

The team tried two things that many people think should help, but they actually made the scores go down.

1. The Reference Sheet (OCR): They tried feeding the computer a text transcript of the image (using a tool called OCR) to help it read the text. But this backfired. The transcript often had typos, broken formulas, or the wrong order of words. It was like giving a student a reference sheet that was full of errors; it confused the computer more than the image itself.
2. The Crash Course (Fine-Tuning): They tried to "teach" the computer specifically on the exam questions using a method called LoRA fine-tuning. They thought this would make the computer an expert. Instead, it made the computer worse. The paper suggests that the computer was already smart enough to read the images, and trying to retrain it on a small, messy dataset just messed up its natural abilities.

The Takeaway

The big lesson from this paper isn't about inventing a new, super-intelligent brain. It's about inference engineering—which is a fancy way of saying "how we ask the computer to do its job." The authors found that a powerful computer model, when given strict rules, clean inputs, and a good cleanup process, can outperform a slightly smarter model that is allowed to be messy.

They showed that you don't always need to make the AI bigger or train it harder. Sometimes, you just need to tell it to shut up, pick the right letter, and hand in a clean answer sheet. By focusing on these practical details, they turned strong vision-language models into reliable competition winners, proving that sometimes the best way to solve a problem is to manage the output, not just upgrade the engine.