Imagine you just bought a super-smart robot assistant that can talk to you, write stories, and solve math problems. You're excited to let it loose in your neighborhood, but there's a catch: your neighborhood is a giant, bustling market where people speak dozens of different languages, celebrate unique festivals, and have their own deep-rooted traditions. If you only taught this robot using books written in English about life in New York or London, it would be a brilliant scholar but a terrible neighbor. It might accidentally insult a local elder, misunderstand a cultural joke, or worse, give dangerous advice because it doesn't "get" the local rules. This is the world of Large Language Models (LLMs)—AI brains trained on massive amounts of text. The big question scientists are asking right now is: Are these AI brains ready to help people in places like India, where the culture and languages are incredibly diverse?

To answer this, researchers need a way to test the robots. Usually, they use standard tests like "MMLU" or "TruthfulQA," which are like driving tests designed for highways in Europe. But driving in India is different; the roads are narrower, the traffic rules are unique, and the landmarks are totally different. If you only test a car on European highways, you won't know if it can handle a chaotic Indian street market. This paper introduces a new, open-source "driving test" specifically built for India, called Inspect India Evals. It checks if AI models can speak 16 different Indian languages, understand Indian social dynamics (like the complex caste system and religious diversity), and stay safe when asked tricky questions about India's digital ID systems and payment apps.

The Big Test: A New Map for AI

The authors of this paper, working with the Indian government's Ministry of Electronics and Information Technology, realized that the existing AI tests were missing the mark. They built a framework called Inspect India Evals, which is like a six-challenge obstacle course designed specifically for the Indian context. Instead of just asking, "Can you solve this math problem?" they asked, "Can you solve this math problem in Tamil?" or "Can you tell me how to register for a government ID without accidentally teaching someone how to forge it?"

The framework includes six specific challenges:

1. Multilingual MMLU: A knowledge test translated into 16 Indian languages (like Hindi, Bengali, and Tamil) to see if the AI actually understands the facts or just guesses.
2. BharatBBQ: A bias test that checks if the AI holds stereotypes about Indian castes, religions, or regions, rather than just American race or gender issues.
3. DPI Safety: A safety check for India's "Digital Public Infrastructure" (like Aadhaar for ID and UPI for payments). It tests if the AI refuses to help with fraud but still answers helpful questions.
4. Multilingual Safety: Checking if the AI says "no" to harmful requests (like "how to hack a bank") even when asked in a regional language.
5. Jailbreak Resistance: Trying to trick the AI into breaking its rules using multi-step conversations in different languages.
6. Indian Cultural Knowledge: A test of deep cultural facts, from the Constitution to farming policies.

The Race: Who Passed the Test?

The researchers put five different open-source AI models through this obstacle course. These models ranged in size from 8 billion to 32 billion "parameters" (think of these as the number of connections in the AI's brain). They tested models like Sarvam-M 24B (an India-focused model), Gemma 2 27B, Qwen 2.5 32B, DeepSeek-R1 14B, and Llama 3.1 8B.

Here is what they found, and it's a bit of a surprise:

• The Local Hero Wins: The Sarvam-M 24B model, which was specifically tuned for Indian languages and culture, came out on top overall. It scored a 74.4% average. It was especially good at understanding Indian culture (scoring 60.0%) and handling digital safety questions perfectly (100%). It even beat larger, more powerful models on cultural knowledge.
• The Safety Star: Gemma 2 27B came in a very close second with a 72.1% average. It was a perfect safety guard, getting 100% on bias tests and digital safety, though it struggled a bit more with deep cultural facts.
• The Reasoning Robot's Struggle: DeepSeek-R1 14B is famous for being a "reasoning" model that thinks very hard before answering. It aced the factual reasoning test in multiple languages, scoring 80.6%. However, it failed miserably at safety and cultural knowledge, scoring only 20% on digital safety and 10% on cultural facts. It was so focused on "thinking" that it forgot to be safe or culturally aware.
• The Big Models Didn't Win: Interestingly, having more parameters (a bigger brain) didn't guarantee a win. The Qwen 2.5 32B (a 32-billion parameter model) scored lower than the smaller Sarvam-M 24B in many areas. This suggests that for India, specialized training matters more than just raw size.

The "India Fairness Index"

To make sense of all these numbers, the authors created a single score called the India Fairness Index (IFI). This score combines safety, bias, and factual accuracy into one number.

• Gemma 2 27B got the highest IFI score of 83.1%.
• Sarvam-M 24B followed closely with 76.6%.
• DeepSeek-R1 14B dropped to 65.2%, showing that being smart isn't enough if you aren't safe.
• Llama 3.1 8B scored the lowest at 51.6%.

The Verdict: Safety is Everywhere, Culture is Hard

One of the most interesting findings was that all five models were perfect at refusing harmful requests in multiple languages. They all scored 100% on the "Multilingual Safety" test. Whether asked in English, Hindi, or Tamil to do something bad, they all said "no." This is great news!

However, the models struggled significantly with Indian Cultural Knowledge. The specialized model (Sarvam-M) scored 60%, while the others hovered between 10% and 30%. This suggests that standard global AI training isn't enough; you need specific training on Indian history, festivals, and social structures to get it right.

The authors also noted a trade-off: DeepSeek-R1 showed that models designed for complex reasoning can sometimes be tricked into breaking safety rules (only 40% resistance to jailbreaks), while models like Gemma and Sarvam were much tougher (80% resistance).

What This Means

The paper concludes that we cannot just take an AI trained in the West and drop it into India. It's like trying to drive a Formula 1 car on a muddy village road; it might be fast, but it will get stuck or crash. The results suggest that for AI to be truly useful and safe in India, it needs specialized tuning for local languages and cultures.

The authors are careful to say that these results are based on a "pilot" study with a small number of test questions (only 5 samples per task). While the trends are clear, they suggest that more testing with larger groups of questions is needed to be absolutely certain. But the message is loud and clear: Specialized, culturally aware models are the future of AI in India, and we need better tests to make sure they are safe before we let them loose.

The code and data for this new "driving test" are now open for everyone to use, so developers can keep building better, safer AI for the world's most diverse nation.