RTI-Bench: A Structured Dataset for Indian Right-to-Information Decision Analysis

Imagine India's Right to Information (RTI) system as a massive, bustling library where every citizen has the right to ask the government for any book (information) they want. But here's the catch: the librarians (government officials) often reply in a secret code, and the head librarian (the Central Information Commission or CIC) writes the final verdicts in a dense, legal language that feels like reading a foreign spellbook. Most people give up because they can't tell if an appeal is worth the trouble or what the decision actually means.

This paper introduces RTI-Bench, a new tool designed to translate that "spellbook" into plain English and organize the library's records so computers can learn to help us.

Here is a breakdown of what the paper actually does, using simple analogies:

1. The Problem: A Wall of Text

The CIC issues thousands of decisions every year. They are public, but they are written in a way that assumes you are a lawyer. If you want to know if you should fight a government decision, you have to wade through pages of complex jargon. The paper argues that we need a "Rosetta Stone" to decode these documents.

2. The Solution: RTI-Bench (The Organized Filing Cabinet)

The authors built a structured dataset called RTI-Bench. Think of this as taking a chaotic pile of 1,516 legal documents and turning them into a neat, digital filing cabinet where every folder has clear labels.

• What's inside: Each file has tags for the outcome (Did the citizen win?), the reasons used (Why did they lose?), and the timeline of events.
• The Source: They gathered data from two places:
  1. 1,218 existing summaries found online (which they cleaned up).
  2. 298 fresh PDFs downloaded directly from the CIC website between 2023 and 2026.

3. The Method: The "Robot Librarian" (Rule-Based Extraction)

You might expect them to use a fancy AI to read these documents. Instead, they built a rule-based pipeline.

• The Analogy: Imagine a robot librarian that doesn't "think" or "guess." Instead, it follows a strict checklist: "If you see the word 'DECISION' in all caps, look at the next line. If you see 'Rs.', look for a number."
• Why do this? The authors say using a smart AI (LLM) to label legal documents is risky because the AI might make up facts or get confused. Their "dumb" robot is 100% reproducible, costs nothing, and runs in under 90 seconds on a normal laptop.
• The Result: They checked 50 random files manually and found the robot was 95.3% accurate.

4. The Challenge: Three Different "Fonts"

The paper discovered that the CIC has changed its document templates three times since 2017.

• 2023a: Uses a specific "Facts / Decision" structure.
• 2023b: Separates "Observations" from "Decisions."
• 2026: The current format, which uses big, bold "DECISION" blocks.
  The robot was programmed to recognize these three different "fonts" and switch its reading strategy accordingly, much like a translator switching between different dialects.

5. The First Test: Can a Computer Guess the Winner?

The authors tested a basic AI model (Mistral 7B) on 100 cases to see if it could predict the outcome just by reading the text.

• The Score: The AI got 57.3% accuracy.
• The Comparison: If the AI just guessed the most common answer every time (like a student guessing "C" on a multiple-choice test), it would only get 14.3% right.
• The Takeaway: The AI is doing significantly better than random guessing, proving that computers can learn to understand these legal decisions, but there is still a lot of room for improvement (especially for rare outcomes).

6. What Can We Do With This? (The Four Tasks)

The paper proposes four specific games for researchers to play with this data:

1. Outcome Prediction: "Based on the story, will the citizen win or lose?"
2. Exemption Detection: "Which secret rule (like 'privacy' or 'commercial confidence') did the government use to say 'no'?"
3. Plain-Language Summary: "Translate this legal order into a sentence a normal person can understand."
4. Compliance Check: "Did the government actually follow the order, or do they need to be fined?" (Note: They only have 17 examples of this, so it's just a pilot test).

7. The Fine Print (Limitations)

The authors are very honest about what this tool cannot do yet:

• Coverage: They only have about half of the new PDFs fully labeled; the rest need more work.
• Scope: This only covers the Central Commission, not the state-level commissions (which handle even more cases).
• Missing Logic: The robot only catches exemptions when the government explicitly writes down the rule number. If they use a rule without naming it, the robot misses it.
• Ethics: They warn that this tool shouldn't be used to discourage people from filing appeals. Just because an appeal is "unlikely to win" doesn't mean it shouldn't be filed, as the act of filing itself holds the government accountable.

Summary

RTI-Bench is the first organized, public dataset of Indian RTI decisions. It uses a simple, rule-based robot to turn messy legal PDFs into clean data. Early tests show that computers can start to understand these decisions, paving the way for future tools that might help ordinary citizens navigate the complex world of government information requests.