Technical Summary: A Pāṇinian Foundation for Indic Language Processing

1. Problem Statement

The current Natural Language Processing (NLP) infrastructure for Indic languages (spoken by over a billion people) is critically underdeveloped and structurally fragmented. The field currently organizes tools, datasets, and benchmarks around genealogical language families or individual languages, leading to:

• Redundant Engineering: Separate morphological analyzers, parsers, and annotated datasets are built for each language, often from scratch.
• Incompatible Schemata: Annotation standards (e.g., Universal Dependencies) do not map effectively to the deep grammatical categories used in Indic languages, causing lossy conversions and preventing cross-lingual transfer.
• Ineffective Multilingual Models: While massively multilingual models (MLLMs) achieve broad coverage, they optimize for fluency rather than deep morphosyntactic understanding. They rely on surface-level signals (subwords, shared vocabulary) rather than the underlying structural unity, leading to poor performance on low-resource languages and an inability to handle dialectal or register variations (diglossia).
• Semantic Mismatch: Standard NLP frameworks, rooted in Western linguistic assumptions, treat nouns as static entities. This fails to capture the process-oriented ontology of Indic languages, where nominal meaning is derivationally rooted in verbal roots (dhātu) and organized around event structures.

The core issue is the denial of a deep, shared architectural regularity: over two millennia of convergence around Sanskrit have unified Indic languages under a common morphosyntactic framework formalized in Pāṇini's Aṣṭādhyāyī. Current tools ignore this "structural commons," treating languages as disparate entities rather than variations of a shared metalanguage.

2. Methodology and Theoretical Framework

The paper proposes shifting the computational paradigm from genealogical classification to a Pāṇinian framework. This approach treats the Aṣṭādhyāyī not merely as a historical grammar but as a unifying computational architecture.

• The Pāṇinian Architecture: The authors argue that Pāṃini's system provides a formal metalanguage that defines the "operating system" for South Asian discourse. Key features include:
  • Determinacy: Unlike Chomskyan formalisms that require statistical disambiguation, Pāṇini's rule-precedence mechanisms ensure a single derivation for every grammatical sentence, avoiding overgeneration.
  • Functional Unity: It captures functional and architectural commonalities (morphology, syntax, semantics) that transcend genealogical boundaries (e.g., between Indo-Aryan and Dravidian languages).
  • Process-Oriented Semantics: It models nouns as roles within implicit event structures (kāraka), rather than static labels.
• Leveraging Existing Signals: The methodology relies on the observation that cross-lingual transfer is already stronger within Indic languages than between Indic and non-Indic languages, suggesting models implicitly learn this shared structure. The paper argues for making this implicit learning explicit through grounded benchmarks.
• Benchmark Design: Instead of proposing new models, the paper proposes a four-part benchmark suite designed to render the shared architecture explicit, measurable, and exploitable. These benchmarks are designed to be multilingual from the ground up, moving beyond ad-hoc extensions of single-language resources.

3. Key Contributions

The paper's primary contribution is the proposal of a unified evaluation infrastructure consisting of four thematic benchmark clusters:

5.1 Morphological Segmentation and Tagging

• Goal: Decompose words into Pāṃinian roots (dhātu) and affixes (pratyaya, lakāra) and interpret their semantic functions.
• Scope: Extends existing Sanskrit benchmarks (e.g., MorphTok) to modern Indic languages.
• Tasks:
  • Multilingual segmentation and tagging.
  • Etymological grounding: Identifying the Sanskrit root of a modern Indic word and its core meaning.
  • Cross-lingual word-sense disambiguation based on etymological grounding.

5.2 Syntactic Parsing via Pāṃinian Dependencies

• Goal: Create a unified multilingual dependency parser grounded in kāraka (semantic case) relations rather than Universal Dependencies.
• Scope: Builds upon the AnnCorra project and initial Pāṃinian universal dependency efforts.
• Tasks: Constructing or converting dependency treebanks for multiple Indic languages using Pāṃinian labels and evaluating parsers trained on this unified schema against language-specific baselines.

5.3 Semantic Role Labeling (SRL) and Inference

• Goal: Assess semantic comprehension across registers (literary vs. colloquial) and dialects.
• Scope: Extends kāraka-grounded work from Question Answering (QA) to entailment and inference.
• Tasks:
  • Register Adaptation: Models must preserve semantic content while translating between literary and colloquial forms (e.g., Sanskritized Hindi vs. Khariboli).
  • Dialectal Robustness: Testing performance across dialect continua (e.g., Tamil or Bengali variations) where surface phonology differs but morphosyntactic templates remain invariant.
  • Code-Mixing Generalization: Handling sociolinguistic shifts from formal text to informal speech.

5.4 The Extended Indic "Sprachbund" and Information Disorder

• Goal: Test the limits of the Pāṃinian framework's structural unity and apply it to social challenges.
• Scope:
  • Linguistic Extension: Evaluating transfer to diaspora varieties (e.g., Fiji Hindi), conservative languages (e.g., Romani), and languages connected by civilizational influence rather than genealogy (e.g., Tibetan, Thai, Burmese) which share scripts and phonological taxonomies but lack the full morphosyntactic substrate.
  • Information Disorder: Developing benchmarks to detect cross-lingual misinformation. Models must identify semantically equivalent claims across languages and registers, tracking how narratives mutate while remaining traceable to a shared semantic substrate.

4. Results and Evidence

The paper does not present new experimental results from a single model but synthesizes empirical evidence from existing literature to support the Pāṃinian hypothesis:

• Transfer Learning: Studies (Bafna et al., Nag et al.) show that cross-lingual transfer is significantly stronger between Indic languages than between Indic and non-Indic languages, even when controlling for script differences.
• Morphological Analysis: Models based on Pāṃinian principles (e.g., MorphTok) show downstream improvements in machine translation and named entity recognition.
• Syntactic Parsing: Incorporating kāraka relations has shown immediate improvements in Indic-language QA compared to Universal Dependencies.
• Data Efficiency: Research (Chang et al.) indicates that adding structurally aligned multilingual data improves low-resource modeling more effectively than simply scaling data volume, suggesting the "structural core" is the key driver of performance.
• Historical Convergence: Evidence from Tamil grammarians (e.g., Tolkāppiyam, Vīracōl̲iyam) demonstrates that structural categories similar to Pāṃini's were discovered independently within Dravidian traditions, confirming the architectural unity is not merely a result of Sanskrit imposition but a shared computational primitive.

5. Significance and Claims

The paper claims that adopting a Pāṃinian foundation offers three primary benefits:

1. Accuracy and Efficiency: Systems grounded in this framework would be more accurate and data-efficient by leveraging the shared structural substrate, effectively merging sparse resources into a single high-resource "metalanguage" bedrock.
2. Interpretability: The framework provides a path to mechanistic interpretability. It raises the scientific question of whether neural models trained on Indic languages spontaneously learn internal representations corresponding to Pāṃinian categories (kāraka, dhātu) without explicit supervision. If they do, it suggests Pāṃini's framework captures genuine computational primitives of these languages.
3. Practical Application: A unified approach would enable robust tools for real-world challenges, such as detecting misinformation across India's diverse linguistic landscape and building parsers that generalize across dialects and registers.

The authors conclude that the field has the "ingredients" (isolated advances in morphology, syntax, and semantics) but lacks the "architecture" to connect them. By building benchmarks explicitly grounded in the Aṣṭādhyāyī, the computing community can move beyond statistical recovery of structure to a designed, interpretable system that reflects how Indic languages actually function.