Technical Summary: RAG System for Supporting Japanese Litigation Procedures

Problem Statement

The paper addresses the challenge of deploying Retrieval-Augmented Generation (RAG) systems within the specific context of Japanese medical litigation procedures. While Large Language Models (LLMs) offer potential efficiency gains in legal workflows, their direct application faces critical barriers regarding legal norms and procedural fairness.

In Japanese civil litigation, particularly in medical cases, judges rely on "expert commissioners" (e.g., physicians) to clarify technical disputes. A RAG system intended to substitute or support these roles must adhere to strict adversarial principles:

1. Prohibition of Private Knowledge: Judges cannot rely on knowledge not submitted by the parties or contained in the evidence.
2. Temporal Validity: The standard of care or legal precedent applicable to a case depends on the specific time of the incident versus the time of the trial.
3. Faithfulness: Generated responses must strictly originate from the provided context to avoid hallucinations that could mislead judicial deliberation.

Existing RAG research focuses primarily on accuracy and reducing hallucinations but lacks frameworks for compliance with legal procedural norms, specifically regarding the provenance, validity period, and procedural admissibility of retrieved knowledge.

Methodology and Proposed Framework

The paper proposes a design for a "norm-compliant RAG" system tailored to Japanese medical litigation. Rather than presenting experimental results, the authors define a theoretical framework and a set of three core requirements that such a system must satisfy. The system architecture (illustrated in Figure 2) integrates a retrieval module with a timestamp maintenance mechanism and an LLM generator.

The methodology is structured around three specific technical requirements:

1. Controlling Knowledge Sources (Procedural Compliance)

To satisfy the adversarial principle, the system must restrict external knowledge to sources that are procedurally admissible.

• Mechanism: The retrieval scope is strictly limited to documents that have undergone critical verification by expert communities and are equally accessible to both parties.
• Implementation Strategy: The system filters retrieval results, labeling outputs derived from sources outside the predefined, admissible scope as inappropriate. This ensures the system does not introduce "private knowledge" that would infringe on procedural rights.

2. Attribution and Faithfulness

To ensure responses are not hallucinations and rely solely on the provided context:

• Mechanism: The system must distinguish between knowledge derived from the RAG retrieval and knowledge from the LLM's pre-training data.
• Implementation Strategy:
  • Data Attribution (DA): The system analyzes whether claims in the generated response originate from the retrieved context or the pre-trained model. Responses relying on pre-training data are flagged as inappropriate.
  • Faithfulness Scoring: The paper references the use of metrics like the Faithfulness Score (e.g., via Ragas), which calculates the ratio of claims in a response supported by the retrieved context against the total number of claims.
  • Verification: The design suggests using explicit constraints in prompting and chain-of-verification steps to ensure candidate sentences are contained within the retrieved context.

3. Issue-Specific Reference Time (Temporal Validity)

The system must manage the "valid time period" of information sources, as the applicable standard of care changes over time.

• Mechanism: The system must recognize that the relevant knowledge for a case depends on the specific issue (e.g., negligence at the time of the act vs. causal relationship at the time of trial).
• Implementation Strategy: The retrieval module must maintain and utilize timestamps and citation networks. It must retrieve knowledge valid at the specific point in time relevant to the dispute, rather than simply retrieving the most up-to-date information. This prevents the system from generating conclusions based on knowledge that did not exist at the time of the alleged incident.

Key Contributions

The paper's primary contribution is the definition of a norm-compliant RAG framework specifically for the Japanese legal domain. Unlike general RAG evaluations that focus on factual accuracy, this work introduces:

1. Procedural Constraints: A formal requirement to filter knowledge sources based on civil litigation norms (admissibility and equal access).
2. Temporal Awareness: A specific architectural requirement to handle the "valid time period" of expert knowledge, acknowledging that legal and medical standards evolve and that the "correct" answer depends on the temporal context of the legal issue.
3. Faithfulness in Legal Context: A shift in evaluation focus from general factuality to source attribution, ensuring that the system does not rely on its internal pre-trained knowledge when generating legal arguments.

Results and Claims

The paper is a preprint presenting a design proposal and theoretical framework; it does not report experimental results, quantitative evaluations, or a deployed system.

• Current Status: The authors state they are currently developing the system and that their future work involves proposing specific methods to satisfy the identified requirements, refining evaluation metrics, implementing the system, and conducting experiments.
• Significance: The authors claim that their work addresses a central, unmet challenge in AI for law: the lack of research focusing on compliance with legal norms and the appropriateness of knowledge sources. By defining these three pillars (source control, faithfulness, and temporal validity), the paper aims to provide the necessary foundation for building RAG systems that can be trusted in high-stakes judicial environments where procedural fairness is paramount.

In summary, the paper argues that for RAG to be viable in Japanese litigation, it must evolve from a tool that simply retrieves "relevant" information to one that retrieves "procedurally admissible, temporally accurate, and strictly faithful" information.