Pramana: Fine-Tuning Large Language Models for Epistemic Reasoning through Navya-Nyaya

This paper introduces Pramana, a novel fine-tuning approach that enhances large language models' epistemic reasoning and reduces hallucinations by training them on the structured, 2,500-year-old Indian Navya-Nyaya logical framework, which enforces a rigorous six-phase reasoning process for evidence-based justification.

The Gist

Pramana: Teaching AI to Think Like an Ancient Philosopher

Imagine you have a brilliant, hyper-fast student who can write beautiful essays and tell amazing stories. But there's a catch: this student is a confident liar. If you ask them a math problem, they might solve it perfectly for the first five steps, then suddenly hallucinate a wrong answer on step six, and insist they are 100% right. They don't actually "know" the answer; they are just guessing based on patterns they've seen before.

This is the current state of Large Language Models (LLMs). They are fluent but lack systematic reasoning.

This paper, titled "Pramana," proposes a radical solution: instead of trying to teach AI new math or coding skills, the authors decided to teach it 2,500-year-old Indian logic. Specifically, they used a framework called Navya-Nyaya to force the AI to think step-by-step, check its own work, and admit when it doesn't know something.

Here is the breakdown of how they did it, using simple analogies.

---

1. The Problem: The "Confident Guessing" Machine

Current AI models are like a parrot that has read every book in the library. If you ask, "How many 'r's are in 'strawberry'?", the parrot might say "3" because it sounds right, even if it's wrong.

Researchers found that if you add a silly, irrelevant sentence to a math problem (e.g., "Alice has 5 apples and likes the color purple"), the AI's performance crashes. Why? Because the AI isn't actually reasoning; it's just matching patterns. When the pattern changes, the AI gets lost.

The Goal: We need an AI that doesn't just guess, but proves its answer.

2. The Solution: The "Six-Step Detective"

The authors introduced a method called Pramana (which means "valid source of knowledge" in Sanskrit). They fine-tuned AI models to follow a strict 6-phase reasoning process derived from ancient Indian philosophy.

Think of this not as a math class, but as a Detective Training Academy. Every time the AI solves a problem, it must act like a detective following a strict protocol:

Phase 1: Samshaya (The "Wait, What?" Moment)

• The Metaphor: Before a detective starts solving a crime, they must admit, "I am confused. I don't know who did it yet."
• What the AI does: It must explicitly state what is uncertain. It cannot jump to a conclusion. It has to say, "I am unsure because the clues are conflicting."

Phase 2: Pramana (The "Evidence Board")

• The Metaphor: A detective pins photos and notes to a corkboard. They can't just say "I feel like the butler did it." They must point to a specific clue.
• What the AI does: It must list its sources of knowledge:
  • Direct Perception: What is explicitly stated in the problem?
  • Inference: What can I logically deduce?
  • Comparison: Is this like a case I solved before?
  • Testimony: What are the universal rules (like math laws) I am using?
  • Crucial Rule: If the AI makes a claim without a "pin on the board," it's hallucinating.

Phase 3: Pancha Avayava (The "Five-Part Argument")

• The Metaphor: This is the formal courtroom presentation. You can't just say "The butler is guilty." You must build a case:
  1. Thesis: "The butler did it."
  2. Reason: "Because he was holding the candlestick."
  3. Universal Rule: "Whenever someone holds the murder weapon, they are the suspect."
  4. Application: "The butler was holding the candlestick."
  5. Conclusion: "Therefore, the butler is the suspect."
• What the AI does: It forces the AI to connect the dots with a "Universal Rule" (a general law) and a "Concrete Example." This stops the AI from making abstract leaps.

Phase 4: Tarka (The "Devil's Advocate")

• The Metaphor: The detective plays a game of "What if?" They try to prove themselves wrong. "Okay, what if the butler didn't do it? Does that create a contradiction?"
• What the AI does: It assumes the opposite of its conclusion and tries to break its own logic. If the logic holds up, the answer is strong. If it breaks, the AI goes back to Phase 1.

Phase 5: Hetvabhasa (The "Fallacy Police")

• The Metaphor: A senior inspector reviews the detective's work to catch mistakes. "Did you confuse correlation with causation? Did you argue in a circle?"
• What the AI does: It checks itself for five specific types of logical errors (like "The ground is wet, so it must have rained"—ignoring the sprinkler).

Phase 6: Nirnaya (The "Verdict")

• The Metaphor: The judge gives the final ruling.
• What the AI does: It gives the final answer. But here is the magic: It can also say "I don't know." If the evidence isn't enough, it admits uncertainty instead of making up a confident lie.

---

3. The Experiment: Training the AI

The researchers took two AI models (a small one and a medium-sized one) and trained them on just 55 examples of these 6-step detective stories.

The Results were fascinating:

• The "Format" Struggle: The AI sometimes forgot to write the exact headers (like "Phase 5: Fallacy Police"). It was about 40% good at following the format.
• The "Thinking" Success: However, when the AI did finish the steps, 100% of the answers were correct.

The Big Takeaway: The AI learned the content of the reasoning (how to think) even if it was messy about the structure (how to write it down). It internalized the logic.

4. Why This Matters

Imagine a doctor AI.

• Current AI: "You have a headache. You probably have a brain tumor." (Confident, but wrong).
• Pramana AI:
  1. Doubt: "Is this a tumor or a migraine?"
  2. Evidence: "Patient has no vision loss (Direct Perception)."
  3. Rule: "Brain tumors usually cause vision loss."
  4. Test: "What if it is a tumor? Then why no vision loss? Contradiction."
  5. Verdict: "Likely a migraine. I am not 100% sure, but the evidence points here."

5. The Trade-off

There is a cost. This method makes the AI "talk" much more.

• Normal AI: Writes a short answer (300 words).
• Pramana AI: Writes a long, detailed report (3,000 words).

The authors argue this is worth it. In high-stakes fields like law, medicine, or safety, we don't want a fast, confident guess. We want a slow, verified, auditable proof.

Summary

The paper shows that we can teach modern AI to be humble, rigorous, and logical by teaching it the ancient rules of Indian philosophy. It turns the AI from a confident guesser into a careful detective that checks its own work before speaking.

Even though the AI sometimes forgets to use the exact "checklist" headers, it has learned the spirit of the logic: Don't just say what you think; prove why you think it.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) currently suffer from an "epistemic gap": they generate fluent text based on statistical pattern matching but lack systematic reasoning capabilities. This leads to:

• Hallucinations: Confident but unfounded claims.
• Brittleness: Performance degrades significantly (up to 65%) when irrelevant context is added to problems.
• Untraceable Reasoning: Outputs lack justification trails linking conclusions to evidence.
• Correlation-Causation Confusion: Models struggle to distinguish causal relationships from coincidental patterns.

Standard approaches like Chain-of-Thought (CoT) prompting improve accuracy but rely on implicit, opaque patterns learned during pre-training rather than enforcing explicit logical structures. There is a need for a framework that structures reasoning from evidence acquisition to conclusion, with built-in verification mechanisms.

2. Methodology: The Pramana Framework

The authors propose Pramana, a novel approach that fine-tunes LLMs on Navya-Nyaya, a 2,500-year-old Indian epistemological framework. Unlike Western formal logic which separates validity from epistemic grounding, Navya-Nyaya integrates logic with explicit knowledge sources.

The methodology enforces a 6-phase structured reasoning process:

1. SAMSHAYA (Doubt Analysis): Classifying the type of uncertainty (e.g., ambiguity, contradiction) to prevent premature conclusions.
2. PRAMANA (Evidence Sources): Identifying valid knowledge sources:
   • Pratyaksha (Direct Perception/Observation)
   • Anumana (Inference)
   • Upamana (Comparison/Analogy)
   • Shabda (Testimony/Authoritative Rules)
3. PANCHA AVAYAVA (5-Member Syllogism): Constructing formal arguments with:
   • Pratijna (Thesis)
   • Hetu (Reason)
   • Udaharana (Universal Rule + Concrete Example)
   • Upanaya (Application)
   • Nigamana (Conclusion)
4. TARKA (Counterfactual Testing): Verifying conclusions via reductio ad absurdum (assuming the opposite to find contradictions).
5. HETVABHASA (Fallacy Detection): Systematically checking for five specific logical fallacies (e.g., erratic reasoning, contradictory reasons).
6. NIRNAYA (Ascertainment): Distinguishing definitive knowledge from hypotheses, enforcing epistemic humility.

Implementation Details:

• Models: Fine-tuned Llama 3.2-3B (Stage 0) and DeepSeek-R1-Distill-Llama-8B (Stage 1).
• Technique: Supervised Fine-Tuning (SFT) using QLoRA (4-bit quantization) via the Unsloth library for efficiency.
• Data: Small, high-quality datasets (20 examples for Stage 0, 55 for Stage 1) manually verified to ensure strict adherence to the 6-phase structure.
• Architecture: A layered system (CLI, Application, Domain, Infrastructure) with automated validators for structure and optional Z3 SMT solver integration for logical verification.

3. Key Contributions

1. First LLM Fine-Tuned on Navya-Nyaya: Demonstrates that neural networks can learn systematic, 2,500-year-old epistemological methodologies, producing structured outputs with all six phases.
2. Dissociation of Content vs. Structure: The study reveals that models can internalize the content of reasoning (semantic correctness) even when strict format adherence (schema compliance) is imperfect.
3. Open-Source Release: All models, datasets (20–55 examples), training infrastructure, and an interactive Hugging Face demo are released to the community.
4. Epistemological Scaffolding: Provides a framework that forces models to identify evidence sources and verify conclusions before answering, addressing the "illusion of thinking" in current LLMs.

4. Experimental Results

The research was conducted in two stages:

Metric	Stage 0 (Llama 3.2-3B)	Stage 1 (DeepSeek-R1-Distill-8B)
Dataset Size	20 examples	55 examples
Format Adherence	40% (4/10)	40% (4/10)
Semantic Correctness	50% (5/10)	100% (10/10)
Training Time	~19 mins	~17 mins
Final Eval Loss	0.691	0.350

Key Findings:

• Semantic Superiority: Stage 1 achieved 100% semantic correctness (correct answers) despite only 40% strict format adherence. This proves the model learned the reasoning logic even if it occasionally missed a header or field.
• Format Plateau: Format adherence remained at 40% across both stages, suggesting that scaling model size or data volume alone does not solve strict schema compliance; explicit structural penalties or constrained decoding are likely needed.
• Base Model Comparison: Base models showed 0% format adherence and significantly lower semantic correctness, confirming the reasoning capability is learned through fine-tuning.
• Token Overhead: The Nyaya approach requires ~3,200 tokens per solution (10x overhead compared to standard CoT), trading efficiency for interpretability and auditability.

5. Significance and Future Directions

Significance:

• Trustworthy AI: By making reasoning steps explicit and traceable, Pramana enables auditing, debugging, and trust-building, crucial for high-stakes domains (legal, medical, safety-critical).
• Epistemic Rigor: It addresses the fundamental flaw of LLMs conflating belief with knowledge by forcing explicit evidence sourcing (Pramana) and fallacy detection (Hetvabhasa).
• Non-Western Logic in AI: It successfully bridges ancient Indian epistemology with modern neural architectures, offering a viable alternative to Western formal logic for AI reasoning.

Limitations & Future Work:

• Format Adherence: The 40% adherence rate is a bottleneck. Future work (Stage 2) plans to use constrained decoding (GBNF), rejection sampling, and Reinforcement Learning (GRPO) with format-specific rewards to enforce strict schema compliance.
• Domain Expansion: Current evaluation is limited to formal logic (constraint satisfaction, Boolean SAT). Future work will apply the framework to causal reasoning, legal arguments, and medical diagnosis.
• Efficiency: The high token cost will be addressed via hybrid architectures that use standard CoT for simple tasks and full Nyaya structure for complex, high-stakes reasoning.

In conclusion, Pramana validates that LLMs can be taught to reason systematically using ancient epistemological frameworks, achieving high accuracy and interpretability, though strict structural enforcement remains a challenge for future optimization.