Technical Summary: Falkor-IRAC

Problem Statement

Current legal AI systems, predominantly relying on Retrieval-Augmented Generation (RAG) with vector-based semantic similarity, fail to address the structural nature of legal reasoning. The paper argues that legal reasoning is not a semantic search problem but a constrained symbolic reasoning process involving precedent propagation, procedural state transitions, and statute-bound inference.

Standard RAG approaches suffer from three critical structural failures in the legal domain:

1. Semantic vs. Legal Relevance: Lexical similarity does not guarantee legal relevance; a landmark precedent may use different vocabulary than the current case yet remain structurally vital.
2. Hallucination as Liability: In legal contexts, a plausible-sounding but fabricated citation is not merely a quality error but a professional liability that can harm litigants.
3. Lack of Statefulness: Legal reasoning is stateful (dependent on procedural timelines, rights attachment, and time limits), a dimension vector models cannot represent.

These failures are particularly acute in India, where a massive backlog of cases and limited legal aid create a severe access-to-justice gap. Existing tools fail to provide voice-first, hallucination-free explanations of court notices or procedural states.

Methodology: Falkor-IRAC

Falkor-IRAC is a graph-constrained generation framework that decouples generation from verification. Instead of using a knowledge graph merely to improve retrieval (as in GraphRAG), Falkor-IRAC uses the graph as a hard constraint on what the model is permitted to claim.

1. Knowledge Graph Schema

The system ingests judgments from the Supreme Court and High Courts of India into FalkorDB, a graph database supporting hybrid vector-plus-graph queries. The schema extends standard citation networks with three novel layers:

• IRAC Structure: Nodes for Issue, Rule, Analysis, and Conclusion, linked to Cases, Statutes, and Judges.
• Procedural State Transitions: A layer of ProceduralEvent nodes (e.g., BAIL_DENIED, HEARING_HELD) connected by TRIGGERS and PRECEDES relationships. This models the litigation flow as a state machine, allowing the system to answer questions about a litigant's current procedural position.
• Conflict-Typed Relationships: Novel edge types (CONFLICTS_WITH, RESOLVED_BY, NARROWED_BY) that explicitly model doctrinal splits (e.g., coordinate bench disagreements) rather than silently resolving them.

2. Ingestion Pipeline

Judgments are processed in three stages:

1. Text Extraction: PyMuPDF is used for layout-aware cleaning.
2. Structured Extraction: An LLM extracts IRAC components, statutes, precedents, and procedural events into constrained JSON, instructed to leave fields empty rather than fabricate data if confidence is low.
3. Graph Loading: Structured data is loaded into FalkorDB using idempotent MERGE operations.

3. Inference Architecture

The generation pipeline operates as a multi-agent loop:

1. Retrieval Agent: Traverses the graph using five parallel strategies (matter-type, statute, issue-keyword, citation-chain, and conflict detection) to retrieve candidate precedents.
2. LLM Synthesis: Generates an answer grounded only in the retrieved candidates.
3. Verifier Agent (Falsifiability Oracle): This is the core innovation. It checks the generated answer against the graph:
   • Verifies the existence of cited cases.
   • Checks for overruling (OVERRULES) or conflict relationships.
   • Validates statute repeal status.
   • Output: Returns VALID, INVALID, CONFLICT, or STALE.
   • Constraint: If the path is INVALID or STALE, the LLM is prompted to revise (up to two attempts). If it fails again, the system abstains rather than returning an unverified answer.

Key Contributions

1. IRAC Knowledge Graph Schema: The first schema for Indian legal reasoning to model litigation flow (procedural state transitions) alongside doctrinal reasoning and conflict-typed precedent relationships.
2. Graph-Constrained Generation: An architecture where LLM outputs are accepted only if a valid supporting path exists in the graph. This shifts the paradigm from "better retrieval" to "external examiner with veto power."
3. Conflict Detection as First-Class Output: The system explicitly surfaces doctrinal conflicts (e.g., unresolved coordinate bench disagreements) rather than resolving them silently, providing honest uncertainty to the user.
4. Graph-Native Evaluation Suite: A proposal to replace lexical metrics (BLEU/ROUGE) with structural metrics: Citation Grounding Accuracy, Path Validity Rate, Hallucinated Precedent Rate, and Conflict Detection Rate.

Results

The system was evaluated on a proof-of-concept corpus of 51 Supreme Court judgments (spanning 1949–2026) using 10 queries across bail, constitutional law, service law, and contempt.

• Verifier Accuracy: The Verifier Agent correctly validated all 8 real Supreme Court citations submitted (100% grounding accuracy) and correctly rejected 2 fabricated citations (0% hallucination rate on fabricated input).
• Generation Performance: Using Mistral 7B-Instruct on consumer CPU hardware, 3 out of 10 queries completed within the 300-second timeout. The single completed query with structured citations achieved 100% citation grounding accuracy.
• Limitations: 7 queries timed out due to CPU inference latency. Bail queries specifically suffered from sparse graph coverage in the small corpus.
• Baseline: No comparison against Vector RAG or GraphRAG was performed in this study; the focus was on architectural validation.

Significance and Claims

The paper positions Falkor-IRAC not as a production-ready system, but as a research prototype demonstrating that legal reasoning is structurally closer to constrained graph traversal than to fuzzy similarity search.

• Honesty over Confidence: The system prioritizes "honest uncertainty" (abstention or conflict flagging) over confident hallucination, which is argued to be more useful for legal aid practitioners and pro se litigants.
• Stateful Reasoning: By modeling procedural state transitions, the system addresses a gap in existing legal AI that treats judgments as static documents rather than traversal records of a legal process.
• Generalizability: The authors argue this architecture (symbolic graph constraints on probabilistic generation) is applicable to any domain requiring traceable, auditable reasoning, such as compliance, patent analysis, and clinical guidelines.

The paper concludes that making the implicit graph of legal reasoning explicit and using it to constrain generation is the necessary direction for trustworthy legal AI, particularly in high-stakes, high-caseload jurisdictions like India.