Technical Summary: LINGUA-AUDIT

Problem Statement

As Large Language Models (LLMs) are increasingly deployed as conversational interfaces for public services and civic welfare schemes in multilingual societies, a critical challenge has emerged: AI-mediated information inequality. This phenomenon is defined as the systematic divergence in the substantive, factual, and actionable information provided to citizens seeking identical information needs across different linguistic conditions.

In high-stakes civic contexts (e.g., healthcare eligibility, statutory documentation, welfare schemes), performance disparities are not merely stylistic or fluency-based. Instead, they manifest as substantive harms where non-English or code-switched (e.g., Hinglish) queries yield incomplete, outdated, or factually erroneous guidance compared to English queries. Existing multilingual evaluations largely focus on surface-level capabilities (perplexity, translation scores, or multiple-choice exams) and fail to assess whether identical civic information needs are met equitably across languages.

Methodology

The paper presents LINGUA-AUDIT, a reproducible framework designed to systematically evaluate substantive information inequality. It is a benchmark-design and measurement protocol, not an empirical comparison of live models.

1. Benchmark Design and Task Construction

• Scope: The framework anchors 150 real-world administrative tasks across five essential civic domains in India: Healthcare, Education, Public Services, Finance, and General Knowledge.
• Ground Truth: Each task is grounded in public statutory authorities recorded in a master task registry (sourced from official guidelines, gazette notifications, ministry circulars, and departmental domains). However, the ground truth represents internal curation by the project author based on official administrative schemes, rather than an independently certified legal gold standard. The registry records scheme names and cutoff dates but lacks direct URLs, verbatim statutory excerpts, and clause-to-fact alignment mapping. Tasks are indexed by Underlying Information Needs (UINs), which are language-agnostic conceptual specifications of communicative intent and factual scope, eliminating English-normative bias.
• Fact Structure: Every task includes:
  • Critical Facts (CF): 3 mandatory statutory criteria per task (e.g., income ceilings, deadlines).
  • Important Facts (IF): 2 essential procedural contexts per task.
  • Safety Constraints: 1 explicit boundary per task to prevent financial exploitation.

2. Experimental Configuration

The framework defines a complete factorial matrix for data collection and scoring:

• 150 Tasks × 3 Language Conditions (Formal English, Devanagari Hindi, Romanized Hinglish) × 4 Target Model Slots (Google, OpenAI, Anthropic, Meta) × 3 Repetitions.
• Total Cells: 5,400 factorial cells.
• Current Status: The paper reports on a deterministic software fixture (synthetic baseline) generated in-memory to verify pipeline integrity. No live model API queries have been executed yet; the 5,400 records are programmatic fixtures, not genuine model outputs.

3. Measurement Metrics

The framework operationalizes 13 response-quality metrics across five dimensions:

• Factual Precision: Claim-level precision (M1) and contradiction-only precision (M1b).
• Completeness & Omission: Critical Fact Coverage (M2a), Weighted Completeness (M2c), and Critical Fact Omission Rate (M3).
• Factual Errors: Deduplicated error counts (M4a), binary error indicators (M4b), and normalized error rates (M4c).
• Authenticity & Safety: Citation quality grades (M5), domain safety levels (M6), and semantic preservation relative to UINs (M7).
• Stochastic Consistency: Completeness variance (M8a) and Critical Fact Jaccard Concordance (M8b) across repetitions.

4. Statistical Analysis Plan

A prespecified statistical plan utilizes:

• Generalized Linear Mixed Models (GLMMs) for proportional outcomes (Binomial for M1, M2a, M3; Negative Binomial for M4a).
• Cumulative Link Mixed Models (CLMMs) for ordinal ratings (M5, M6, M7).
• Two-Part Hurdle models for variance and concordance metrics.
• Multiplicity Control: Family-wise error rate control via Holm–Bonferroni step-down correction for a primary confirmatory family of 9 hypotheses (3 endpoints × 3 language contrasts).

Key Contributions

1. Civic-Information Benchmark Design: A curated set of 150 administrative tasks indexed by language-agnostic UINs, moving beyond translated corpora to test authentic localized problem-solving.
2. Multi-Dimensional Operational Metrics: Mathematical specifications for 13 metrics that specifically target substantive information inequality (omissions, hallucinations, safety risks) rather than surface fluency.
3. Methodological Governance: Explicit quarantine of invalid prior synthetic evaluator validation artifacts (where automated evaluators were cloned to serve as "human" ground truth) and a clear separation of software correctness from empirical construct validity.
4. Reproducible Software Pipeline: An open-source Python scoring architecture verified by 115 passing unit and regression tests against the 5,400-cell factorial matrix.
5. Code-Switching Integration: Treating Hindi-English code-switching (Hinglish) as a primary, first-class evaluation condition alongside formal English and Hindi.

Results and Validation Status

The paper explicitly states that no empirical model comparisons or language disparity rankings are reported. The results presented are strictly technical validations of the framework itself:

• Pipeline Integrity: The software successfully generated and verified 5,400 synthetic records with 100% coverage across the factorial matrix.
• Cryptographic Verification: The dataset fixture file was cryptographically audited with verified SHA-256 hashes.
• Baseline Diagnostics: Synthetic fixtures showed zero latency and zero variance across repetitions, confirming the pipeline's ability to parse and serialize data correctly.
• Validation Limitations: All 13 quality metrics remain empirically unvalidated pending genuine bilingual human annotation. A prior automated evaluator validation was found to rely on cloned synthetic references and was quarantined.

Significance and Claims

The paper positions LINGUA-AUDIT as a necessary infrastructure for algorithmic governance in multilingual societies. Its significance lies in:

• Shifting the Evaluation Paradigm: Moving from "surface fluency" to "substantive information equity," recognizing that in civic contexts, a missing mandatory document or an outdated deadline constitutes a tangible administrative harm.
• Preventing Circular Validation: By exposing and quarantining a "circular self-validation" error (where synthetic outputs were mistaken for human ground truth), the framework establishes a rigorous standard for methodological transparency in AI auditing.
• Future-Ready Infrastructure: The framework provides the prespecified measurement instruments and statistical architecture required to audit frontier models once live data collection and human evaluator calibration are completed.

The author concludes that LINGUA-AUDIT does not yet answer whether specific models exhibit inequality; rather, it provides the open, reproducible, and statistically rigorous framework required to detect and quantify such inequality once the necessary empirical data (live model responses and human annotations) are gathered.