ArogyaSutra: A Multi-Agent Framework for Multimodal Medical Reasoning in Indic Languages

This paper introduces ArogyaSutra, a multi-agent framework and a large-scale multilingual multimodal medical dataset designed to enhance AI-driven healthcare reasoning in low-resource Indic languages by integrating tool grounding, dual-memory mechanisms, and actor-critic distillation.

The Gist

Imagine a world where a patient in a rural village in India can show a doctor a photo of a rash or an X-ray and ask, "What is wrong with me?" in their local language, like Hindi, Bengali, or Tamil. Currently, the most advanced AI doctors are like brilliant scholars who only speak English and Chinese. If you ask them a complex medical question in a local Indian language, they often get confused, give short answers, or make up facts because they haven't been trained on how to "think" through a problem in those languages.

This paper introduces a solution called ArogyaSutra (which translates roughly to "The Health Thread") and a massive new library of medical questions called ArogyaBodha ("Health Knowledge").

Here is how it works, broken down into simple concepts:

1. The Problem: The "One-Size-Fits-None" AI

Think of current medical AI models as a single, very smart student who has read every English medical textbook but has never visited a village in India.

• The Issue: If you show this student a picture of a broken bone and ask in Hindi, "What is this?" they might struggle to connect the image to the words. They often just guess the answer without explaining why, or they switch to English, which the patient might not understand.
• The Gap: There was no big "textbook" of medical questions and answers in Indian languages that included pictures. Without this, the AI couldn't learn how to reason step-by-step in those languages.

2. The Solution: Building the Library (ArogyaBodha)

First, the researchers built a massive new library called ArogyaBodha.

• The Analogy: Imagine gathering 40,000 flashcards from eight different sources (like medical exams, image databases, and textbooks).
• The Work: They took these cards, which were mostly in English, and carefully translated them into seven major Indian languages (like Hindi, Tamil, and Marathi).
• The Quality Check: They didn't just use a robot to translate. They had real doctors check the cards to make sure the medical meaning wasn't lost. They also used a "reverse translation" test (translating Hindi back to English) to ensure the meaning stayed the same.
• The Result: A huge, high-quality dataset covering 31 body systems (like the heart, lungs, skin) and 21 medical fields, all in local languages with pictures.

3. The New AI Doctor: ArogyaSutra

Instead of just giving the AI a textbook and asking it to memorize answers, the researchers built a team of two AI agents that work together like a Senior Intern and a Supervising Professor.

• The Actor (The Intern): This is the AI that looks at the patient's photo and the question.

  • The Superpower: The Intern doesn't just stare at the picture. It has a toolbox. If the picture is blurry, it can "zoom in." If it needs to find a specific edge, it can use an "edge detector." It acts like a detective using magnifying glasses to find clues.
  • The Memory: The Intern has two notebooks:
    • Short-term memory: Remembers the mistake it just made in the last second.
    • Long-term memory: Remembers patterns of mistakes it has made over the whole conversation.
  • The Action: Instead of guessing the answer immediately, the Intern writes down its thinking steps.

• The Critic (The Professor): This AI reads the Intern's notes and the clues found.

  • The Job: It checks: "Is the medical logic correct?" and "Is the language natural?"
  • The Feedback:
    • If the Intern made a language mistake (e.g., using the wrong grammar), the Professor corrects it in English to stabilize the thinking.
    • If the Intern made a medical logic mistake (e.g., confusing a tumor with a cyst), the Professor corrects it in the local language (like Hindi) so the context is clear.
  • The Loop: If the answer is wrong, the Professor sends it back to the Intern with a note: "Try again, but remember this mistake." They repeat this until the answer is perfect.

4. The "Code-Switching" Trick

Sometimes, the AI gets stuck because the local language is too complex for its current logic. The system has a clever trick called Code-Switching.

• The Analogy: Imagine you are trying to solve a hard math problem in a language you are still learning. You might switch to English for the complex math part, then switch back to your native language to explain the answer.
• How it helps: The system automatically switches between English (for strict logic) and the local Indian language (for communication) to ensure the reasoning stays sharp while the answer remains understandable.

5. The Results: A Smarter, More Reliable AI

The researchers tested this new "Intern-Professor" team against other AI models (including the best ones from Google and OpenAI).

• The Outcome: ArogyaSutra consistently scored higher than all other models in every Indian language tested.
• The Proof: Even when the AI was tested on medical questions it had never seen before (like a new type of X-ray), it still performed better than the others.
• The Key Takeaway: By using the tool-based vision (zooming in), the memory system (remembering past mistakes), and the two-agent loop (Intern + Professor), the AI learned to "think" step-by-step rather than just guessing.

Summary

The paper claims that by creating a massive, verified library of medical questions in Indian languages and teaching an AI to use a "think-aloud" method with a partner who corrects its mistakes, they have built a system that can reason through medical problems in local languages much better than anything that exists today.

Important Note: The paper emphasizes that this is a research framework to improve reasoning accuracy. It does not claim the AI is currently ready to replace human doctors in hospitals, but rather that it is a significant step toward making AI healthcare tools fair and reliable for non-English speakers.

Technical Summary

Technical Summary: ArogyaSutra

Problem Formulation

Multimodal Large Language Models (MLLMs) have demonstrated reasoning capabilities in general domains but face significant limitations in specialized healthcare settings, particularly within multilingual and low-resource contexts. Existing medical MLLMs are predominantly English-centric and rely on direct-prediction paradigms that fail to capture the interleaved image–text reasoning required in clinical scenarios. Furthermore, current benchmarks are largely monolingual, text-centric, and closed-form, offering limited insight into multilingual multimodal reasoning. This gap results in degraded logical fidelity and unstable language behavior when models attempt to process complex medical queries in Indic languages, hindering equitable access to AI-driven healthcare assistance in regions like rural India.

Methodology

1. Dataset: ArogyaBodha

To address the lack of high-quality benchmarks, the authors introduce ArogyaBodha, a large-scale multilingual multimodal medical question-answer dataset.

• Composition: The dataset contains 40,857 samples covering seven major Indian languages (Bengali, Hindi, Assamese, Tamil, Telugu, Punjabi, Marathi) and English.
• Sources: Curated from eight heterogeneous sources, including benchmark datasets (MedXpertQA, MedTrinity-25M, PMC-VQA, etc.), medical imaging repositories, and Indian postgraduate medical entrance examinations (NEET-PG, FMGE).
• Scope: It spans 31 body systems, six imaging modalities, and 21 clinical domains.
• Curation: Questions are generated using few-shot prompting with GPT-4o-mini, filtered for reasoning depth and image-question relevance using Gemini-2.5-Flash, and verified by medical practitioners.
• Translation: High-quality translations into Indic languages are performed using Gemini-2.5-Pro. Translation quality is validated via reverse-translation analysis (BLEU4 and Cosine Similarity) and manual review by medical practitioners, ensuring clinical meaning preservation.

2. Framework: ArogyaSutra

The authors propose ArogyaSutra, an actor-critic-based multi-agent framework designed for step-wise, reasoning-aware decision making.

• Architecture: The framework utilizes a dual-agent system (Actor and Critic) instantiated from the same multimodal backbone (Qwen-VL-2.5-7B).
• Visual Grounding: The Actor employs a toolkit of four specific vision tools to extract clinically relevant evidence: open-vocabulary object detection, zoom/crop, edge detection, and depth analysis.
• Dual-Memory Mechanism:
  • Short-term Memory: Captures the most recent prediction error and feedback.
  • Long-term Memory: Summarizes prior reasoning steps, linguistic formulations, and accumulated error patterns.
• Actor-Critic Interaction:
  • The Actor processes medical visual inputs and Indic-language queries, invoking grounding tools and predicting intermediate semantic reasoning steps.
  • The Critic evaluates the Actor's output for medical correctness and language consistency.
  • Reflective Feedback: If an error is detected, the Critic diagnoses the source. Language-related errors trigger code-switching (translating the query to English for stabilization), while reasoning errors are addressed by incorporating summaries from memory modules. The feedback is delivered in the corresponding Indic language or English as appropriate.
• Training Strategy: The framework employs code-switched reasoning chain distillation. After simulating Actor-Critic rollouts, the Critic's corrective signals are used to create a refined dataset. The Actor is then fine-tuned via supervised learning to internalize these corrections, allowing it to perform reasoning without invoking the Critic during inference.

Key Contributions

1. Problem Formalization: The paper formalizes the task of multimodal medical reasoning in Indic languages, requiring step-by-step clinical reasoning over images and native language queries to produce linguistically consistent answers.
2. Benchmark (ArogyaBodha): Introduction of a large-scale, expert-verified dataset covering seven Indic languages and diverse clinical domains, enabling reliable evaluation of reasoning accuracy and cross-lingual generalization.
3. Framework (ArogyaSutra): A novel actor-critic framework integrating tool-based visual grounding with dual-memory mechanisms. It explicitly tracks and corrects reasoning errors and linguistic inconsistencies through iterative refinement and adaptive code-switching.
4. Evaluation: Comprehensive experiments demonstrating the framework's superiority over strong baselines in both reasoning accuracy and multilingual alignment.

Results

Experiments were conducted on a test set of 910 samples (130 per language) and an out-of-distribution (OOD) benchmark derived from Spanish residency exams.

• Performance vs. Baselines: ArogyaSutra (based on Qwen2.5-VL-7B) achieved an average accuracy of 43.40%, outperforming the strongest baseline (GPT-4.0 at 39.30%) by +4.1 points. It also significantly surpassed medical-specific models like BioMistral-7B (23.83%) and MedGemma-4B-it (36.11%).
• Improvement over Base Model: The framework improved the base Qwen2.5-VL-7B-Instruct model by +9.2 points, demonstrating the efficacy of the proposed agentic approach.
• Out-of-Distribution (OOD) Generalization: On the OOD dataset, ArogyaSutra achieved 50.4% accuracy, significantly outperforming its base model (35.0%) and other general-purpose or medical MLLMs.
• Ablation Studies:
  • Removing the Critic agent reduced performance to ~33.5%.
  • Removing tool grounding caused the largest performance drop (16.54%), highlighting the necessity of visual tools.
  • Disabling memory mechanisms reduced accuracy by 12.83%.
  • Disabling code-switching also resulted in a significant performance decline (16.54%).

Significance and Impact

The paper claims that ArogyaSutra and ArogyaBodha support equitable healthcare access by enabling multilingual multimodal medical reasoning in low-resource Indic languages. The work aligns with "AI for Healthcare" and "AI for Social Good" by promoting clinically grounded and inclusive medical AI systems. By addressing the failure modes of degraded logical fidelity and unstable language behavior in Indic languages, the framework offers a pathway toward more trustworthy diagnostic explanations for diverse linguistic communities. The authors emphasize that their approach moves beyond simple direct prediction to structured, reflective reasoning, which is essential for real-world applicability in diverse linguistic settings.

Limitations

The authors acknowledge several limitations:

• The model may still produce reasoning errors or misinterpret visual evidence in rare or atypical clinical cases.
• The dataset covers seven major languages but does not capture the full linguistic diversity of India, including low-resource dialects and code-mixed languages common in clinical practice.
• The framework's reliability depends on the underlying visual grounding tools and the base multimodal backbone; failures in perception or error attribution can propagate despite the feedback loop.